Tuesday, August 9, 2022
spot_imgspot_img

A POLICYMAKER ’S TOOLKIT Natasha Kapil Juan Douglas Rogers Christopher Haley Alexandru Florin Ghita Zoe Cordelia Lu Can Arslan A POLICYMAKER ’S TOOLKIT Natasha Kapil Juan Douglas Rogers Christopher Haley Alexandru Florin Ghita Zoe Cordelia Lu Can Arslan © 2021 International Bank for Reconstruction and Development / The World Bank 1818 H Street NW Washington DC 20433 Telephone: 202-473-1000 Internet: www.worldbank.org This work is a product of the staff of the World Bank with external contri- butions. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of the World Bank, its Board of Executive Directors, or the governments they represent. The World Bank does not guarantee the accuracy, completeness, or cur- rency of the data included in this work and does not assume responsibil- ity for any errors, omissions, or discrepancies in the information, or liabil- ity with respect to the use of or failure to use the information, methods, processes, or conclusions set forth. The boundaries, colors, denomina- tions, and other information shown on any map in this work do not imply any judgment on the part of the World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Nothing herein shall constitute, or be construed or considered to be a limitation upon or waiver of the privileges and immunities of the World Bank, all of which are specifically reserved. This report was produced with the financial support of the European Union. Its contents are the sole responsibility of the authors. The views expressed herein can in no way be taken to reflect the official opinion of the European Union. Rights and Permissions The material in this work is subject to copyright. Because the World Bank encourages dissemination of its knowledge, this work may be reproduced, in whole or in part, for noncommercial purposes as long as full attribution to this work is given. Any queries on rights and licenses, including subsidiary rights, should be addressed to World Bank Publications, The World Bank Group, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-2625; e-mail: pubrights@ worldbank.org. Attribution Please cite the work as follows: Natasha Kapil, Juan Douglas Rogers, Christopher Haley, Alexandru Florin Ghita, Zoe Cordelia Lu, Can Arslan. (2022) Scaling Up Romania: A Policymaker’s Toolkit. Washington, D.C. The World Bank. TABLE OF CONTENTS Acknowledgments   8 Introduction    9 Table of Acronyms   11 Executive Summary    12 PART ONE  AN ANALYSIS OF ROMANIA’S POLICY MIX AND FUNCTIONALITY OF ITS ENTREPRENEURSHIP PROGRAMS    22 CHAPTER 1 Overview    23 1.1 Definition of “Instrument”   25 CHAPTER 2  Policy Mix Analysis     26 2.1 Purpose   27 2.2 Method   27 2.3 Findings    28 2.3.1 Budget   28 2.3.2 Objectives   29 2.3.3 Types of Interventions   32 2.3.4 Types of Beneficiaries   33 2.3.5 Firm Characteristics    33 CHAPTER 3  Functional Analysis   35 3.1 Purpose   36 3.2 Method   36 3.3 Overview of Findings   38 3.3.1 Design   38 3.3.2 Implementation   39 3.3.3 Governance   40 3.3.4 Evidence from Variation    40 CHAPTER 4 Conclusions   43 CHAPTER 5 Recommendations to Improve Policy Mix and Function- ality of Entrepreneurship Programs    47 PART TWO  EVIDENCE- AND STAKEHOLDER-DRIVEN POLICY RECOMMENDATIONS   50 CHAPTER 6  Overview of Policy Recommendations   51 Recalibrate the policy mix for starting and scaling high quality innovative firms by (a) Improving the functionality of instruments, and (b) Implementing a comprehensive package of reforms tailored to high quality innovative firms   55 Reform regulations to strengthen entrepreneurship AND investments   59 Establish a one-stop agency (OR “ecosystem hub”)   67 Strengthen ecosystem enablers   71 Create a startup fund   74 Improve entrepreneurship education and strengthen the role of universities in the ecosystem   76 Implement a startup visa program   82 Build and promote a network of Romanian founders and diaspora   85 Scale up through exports   88 Incentivize innovation to foster knowledge spillovers into the private sector   91 Promote the digital economy   96 References   102 ANNEX A Top 12 Interventions Identified by the Romanian Entrepreneurial Ecosystem   103 ANNEX B Instruments Examined   105 ANNEX C Data Classification   107 ANNEX D Detailed Findings of the Functional Analysis   109   ANNEX E List of Countries with a Startup Visas and/or Digital Nomad Visas    125 FIGURES FIGURE 0.1 European Innovation Scoreboard FIGURE A.D.3 Distribution of Portfolio Relationship Index 2021   13 Scores   110 FIGURE 0.2 EIDES Scores Against Change in FIGURE A.D.4 Distribution of Program Objectives Score 2018 – 20   14 Scores   111 FIGURE 0.3 Overlap between Evidence- & FIGURE A.D.5 Distribution of Alternative Stakeholder-Driven Policy Recommendations   20 Instrument Scores   111 FIGURE 1.1  Schematic of the PER Stages   24 FIGURE A.D.6 Distribution of Logic Model Scores   112 FIGURE 2.1 Distribution of Estimated STI Public Budget Allocation in Romania by Top-level FIGURE A.D.7 Distribution of Inputs Scores   112 Objective, 2014–20, in € Million and %   28 FIGURE A.D.8 Distribution of Activities Scores   113 FIGURE 2.2 Cumulative Annualized Budget by FIGURE A.D.9 Distribution of Products/Outputs Number of Instruments   29 Scores   113 FIGURE 2.3 Instruments by General (Top-Level) FIGURE A.D.10 Distribution of Main Beneficiaries Objective   29 Scores   114 FIGURE 2.4 Number of General Objectives FIGURE A.D.11 Distribution of Selection Criteria Supported by Instruments   30 Scores   114 FIGURE 2.5 Instruments by Specific (Intermediary) FIGURE A.D.12 Distribution of Audiences Scores   114 Objective   30 FIGURE 2.6 Instruments by Specific Objective FIGURE A.D.13 Distribution of Outcomes/Impacts (Sole versus Mixed)   31 Scores   115 FIGURE 2.7 Annual Budget by General Objective   31 FIGURE A.D.14 Distribution of M&E (Design) FIGURE 2.8 Annual Budget by Specific Objective   31 Scores   115 FIGURE 2.9 Number of Instruments by Type of FIGURE A.D.15 Distribution of Learning Scores   116 Intervention   32 FIGURE A.D.16 Distribution of Solicitations FIGURE 2.10 Estimated Annual Budget by Type of Scores   116 Intervention   32 FIGURE A.D.17 Distribution of Eligibility Criteria FIGURE 2.11 Number of Instruments by Type of and Application Information   117 Direct Beneficiary   33 FIGURE A.D.18 Distribution of Selection Processes FIGURE 2.12 Estimated Annual Budget by Scores   117 Beneficiary   33 FIGURE A.D.19 Distribution of Program Database FIGURE 2.13 Instruments by Firm Size   34 Scores   118 FIGURE 2.14 Estimated Annual Budget FIGURE A.D.20 Distribution of Program Closure by Firm Size   34 Scores   118 FIGURE 2.15 Instruments by Firm Maturity   34 FIGURE A.D.21 Distribution of Budget Scores   119 FIGURE 2.16 Yearly Average Budget by Stage of Firm Maturity   34 FIGURE A.D.22 Distribution of Organization Quality FIGURE 3.1 Summary of Functional Analysis Scores   119 Indicators   36 FIGURE A.D.23 Distribution of Roles/Autonomy FIGURE 3.2 Average Scores for Design indicators   38 Scores   120 FIGURE 3.3 Average Scores for Implementation FIGURE A.D.24 Distribution of Staff/Training indicators   39 Scores   120 FIGURE 3.4 Average Score for Governance FIGURE A.D.25 Distribution of Incentives Scores   121 Indicators   40 FIGURE A.D.26 Distribution of Process Monitoring FIGURE 3.5 Functional Analysis Results — Scores Scores   121 by Category across the Set   41 FIGURE A.D.27 Distribution of M&E FIGURE 3.6 Average Score and Standard Deviation (Implementation) Scores   122 Across All Instruments   41 FIGURE A.D.28  Distribution of Program FIGURE 6.1 Sequencing of the Policy Coordination Scores   122 Recommendations   52 FIGURE 6.2 Categorization of Policy FIGURE A.D.29  Distribution of Institutions Recommendations   53 Coordination Scores   123 FIGURE 6.3 Local Ecosystems Connectedness   72 FIGURE A.D.30  Distribution of Interaction of FIGURE 6.4  Innovation Performance by Type of Jurisdictions (Awareness) Scores   123 Innovation   92 FIGURE A.D.31  Distribution of Interaction of FIGURE A.D.1 Distribution of Program Origin Jurisdictions (Seriousness) Scores   124 Scores   109 FIGURE A.E.1 International Good Practice FIGURE A.D.2 Distribution of Program Justification Guidelines for Developing an Incentive Scheme Scores    110 Such as a Startup Visa Program   126 TABLES TABLE 0.1 Difference Between Startups TABLE 0.4 Estimated Costs for Proposed Policy and SMEs   12 Recommendations   21 TABLE 0.2 Categorization of Policy TABLE 6.1 Categorization of Policy Recommendations   18 Recommendations   53 TABLE 0.3 Prioritization of policy TABLE 6.2 Estimated Costs for Proposed Policy Recommendations   19 Recommendations   54 8 ACKNOWLEDGMENTS This preparation of this report was led by Natasha Kapil with overall guidance from Ilias Skamnelos. Part One of this report was written by Juan Rogers, Christopher Haley, and Alexandru Ghita, with research and interview sup- port provided by Emanuel Rauta, Robert Cotuna, and Anna Turskaya. Part Two was developed by Christopher Haley and Zoe Lu, with important contributions from Razvan Antonescu, Can Arslan, Marcio Cruz, and Anna Turskaya. The team appreciates the valuable feedback provided by Maja Andjelkovic, Zeneida Hernandez Uriz, Victor Mulas, Justin Hill, and Daniel Querejazu on this and related outputs. Aarre Laakso edited the report. This work was pro- duced in close coordination with the European Commission’s (EC’s) Directorate General for Structural Reform (DG REFORM), Romania’s North-East Regional Development Agency (NE RDA), Rubik Hub, and Romanian authorities, including the General Secretariat of the Government, the National Institute of Statistics, the Ministry of Research, Innovation and Digitization, and the Ministry of Entrepreneurship and Tourism. 9 INTRODUCTION This report was prepared under the Romania Startup Ecosystem Strategy project, funded by the European Commission’s (EC’s) Directorate General for Structural Reform (DG REFORM). This project comes at the request of Romania’s North-East Regional Development Agency (NE RDA) and Rubik Hub and comprises the following in- terlinked deliverables: • Output 1: Report summarizing inputs for an entrepreneurship strategy (aka “National Startup Ecosystem Strategy”) to be developed by the Romanian authorities; • Output 2: Report summarizing policy and instrument toolkit for implementation of the Startup Ecosystem Strategy (this document); • Output 3: (Pivoted) Presentation on startup agencies’ genesis and recommendations for Romania; • Outputs 4 & 5: Notes conceptualizing relevant study visits and value-added partnership options for ecosys- tem events; and • Output 6: Stakeholder consultations and training workshops. This report, “Scaling Entrepreneurship in Romania: A Policymaker’s Toolkit,” presents key considerations for the operationalization of Romania’s first National Startup Ecosystem Strategy and constitutes the abovementioned Output 2. Romania’s entrepreneurship ecosystem remains nascent. To better understand how to accelerate its develop- ment, the World Bank, in partnership with the EC, the NE RDA, and Rubik Hub, launched a suite of activities that included: • Conducting a diagnostic of Romania’s firms and entrepreneurship ecosystem through a combination of pub- licly available data, reports, and original analysis;1 • Conducting a (partial) Public Expenditure Review (PER)2 to evaluate the design and functionality of Romania’s current policy instruments for science, technology, and innovation (STI) with a focus on entrepreneurship; • Engaging Romania’s entrepreneurship ecosystem through a series of Strategy Working Group sessions that culminated in the identification of the “Top 12 Interventions”3; and • Organizing a series of “Ecosystem Showcases” that put a spotlight on the historical and contemporary trajec- tory of peer and aspirational ecosystems, among many other capacity-building activities. Output 1, “Report summarizing inputs for an entrepreneurship strategy (aka the National Startup Ecosystem Strategy)” and this Output 2, “Report summarizing a policy and instrument toolkit for implementation of the National Startup Ecosystem Strategy,” are interconnected reports that present key considerations for the oper- ationalization of Romania’s first National Startup Ecosystem Strategy. Output 1 is an evidence-driven analysis of Romania’s current entrepreneurship ecosystem as well as a study of innovative firm dynamics. It also summarizes a review of Romania’s current policy instruments supporting research, innovation, and entrepreneurship as well as briefly outlines policy recommendations for the entrepreneurship ecosystem. Output 2 presents a mapping of Romania’s current policy mix in support of research, innovation, and entrepreneurship as well as a functional anal- ysis of key programs. Output 2 also presents a more detailed consideration of policy recommendations outlined in 1. See the report “Starting Up: Entrepreneurship Ecosystems in Romania.” 2. See Part 2 of this report. 3. See Annex A of this report for the list of the Top 12 Interventions. Introduction 10 Output 1, in combination with stakeholder-driven interventions. Considered together, these reports combine evi- dence-driven analysis with stakeholder inputs and present the rationale for a comprehensive package of reforms to support high-growth entrepreneurship via the operationalization of the National Startup Ecosystem Strategy. Output 2 (this document) comprises two standalone parts: Part One analyzes the functionality of existing policy instruments. Part Two proposes additional policy recommendations to address deficiencies in the existing policy mix with a view to increasing the entry of high-quality firms and the number of high-growth Romanian firms (that is, firms that successfully scale up). Output 3 presents the different establishment pathways followed by entities supporting startup ecosystems with recommendations for establishment of such an entity in Romania. This builds on the policy recommenda- tion for the establishment of a one-stop agency (or “ecosystem hub”) included in Outputs 1 and 2. This recommen- dation was ranked second by Romanian entrepreneurship stakeholders on their list of the Top 12 Interventions to build the ecosystem. The objective of establishing a one-stop agency is to address the persistent problem of weak institutional capacity and centralize the entrepreneurship agenda under an institution empowered to imple- ment Romania’s first ever National Startup Ecosystem Strategy. Outputs 4 and 5 are presented jointly to conceptualize relevant study visits and value-added partnership options for the Romanian ecosystem. It does through the discussion of historical trajectories of peer and aspi- rational ecosystems to guide Romania’s ecosystem as it evolves. This note also surfaces additional organizations and partnerships with which Romania can promote knowledge transfer on how to build nascent ecosystems in the specific priority areas identified by ecosystem stakeholders. Output 6 is a presentation that summarizes all the stakeholder consultations and training workshops con- ducted under this project. This includes, but is not limited to: • Ecosystem Showcases (8). Featured Ecosystems include New York City, Boston, Helsinki, Israel, Paris, Estonia, Serbia, and Portugal; and • Workshops for the project’s core working group on findings from the World Bank’s flagship reports: High- Growth Firms: Facts, Fiction, and Policy Options for Emerging Economies (Grover Goswami, Medvedev, and Olafsen 2019) and Europe 4.0: Addressing Europe’s Digital Dilemma (Hallward-Driemeier et al. 2020), as well as a workshop on creating a Theory of Change for the National Startup Ecosystem Strategy and “Effective Governance Models for Innovation Agencies” (Aridi and Kapil 2019).. These activities were undertaken to (i) build implementation capacity of the project’s core working group to draft and implement a National Startup Ecosystem Strategy and (ii) deepen international linkages and networks among key actors in Romania’s entrepreneurial ecosystem with their counterparts in peer and aspirational ecosystems. 11 TABLE OF ACRONYMS COVID-19 Coronavirus disease 2019 CTO Chief technology officer CVC Corporate venture capital DESI Digital Economy and Society Index EC European Commission EIDES European Index of Digital Entrepreneurship Systems EIF European Investment Fund EIS European Innovation Scorecard ERDF European Regional Development Fund EU European Union ESOP Employee stock options plan GDP Gross domestic product IB Intermediate body ICT Information and communication technologies IP Intellectual property IT Information technology KPI key performance indicator M&E monitoring and evaluation MA Managing authority MORID Ministry of Research, Innovation, and Digitalization MSMEs Micro, small, and medium enterprises NCBIR Poland’s National Center for Research and Development NE RDA North-East Regional Development Agency OECD Organisation for Economic Co-operation and Development OP Operational programme PER Public Expenditure Review POC Programul Operațional Competitivitate POCU Programul Operațional Capital Uman POR Programul Operațional Regional R&D Research and development SMEs Small and medium enterprises STEM Science, technology, engineering, and mathematics STI Science, technology, and innovation TRL Technology readiness level TTO Technology transfer office VC Venture capital VCs Venture capitalists 12 EXECUTIVE SUMMARY Romanian policymakers include all categorizes of entrepreneurs — including micro, small, and medium enter- prises (MSMEs), as well as necessity and opportunity entrepreneurs — in their considerations of “entrepre- neurship”. This phenomenon of bundling entrepreneurship under the generalist label of MSMEs is not exclusive to Romania and is observed in other countries. However, it is not always helpful from a policy perspective. At the request of the Romanian ecosystem stakeholders, this report focuses on strengthening the policies, programs, and institutions essential for promoting growth-oriented startups, particularly involving technology or digital firms. Our analysis of Romania’s current policy mix demonstrates that policymakers do not clearly distinguish between startups and small and medium enterprises (SMEs) and thus the former remain underserved despite their promise. Startups have many unique characteristics that distinguish them from other firms and SMEs: they are typically young, highly innovative, work with emerging technology, in nascent sectors, and are often still trying to establish a scalable business model. There are fundamental differences between a young startup attempting to scale within a new industry, and an old SME with little growth ambition in a traditional sector: even though they may be the same size, their core characteristics and the resources on which they draw will be quite distinct. For exam- ple, the high technical risk or market risk of many startups means that risk finance such as seed or venture capital is often appropriate, but these will not be interested in established firms with no growth ambition. For these rea- sons, startups are best considered as a specialist subset of SMEs with particular needs, and hence requiring spe- cific policies, programs, and instruments. See table 0.1 below for a side-by-side comparison of startups vs SMEs. TABLE 0.1  Difference Between Startups and SMEs STARTUPS SMEs Aspiration • Scalable business, high growth • Stable growing business, earn profits Growth rate • Faster • More gradual Risk • High technology and market risk • Dominant market risk Funding • Often seek early and large-scale funding through • Self-funded via organic growth or financed from public support, “friends, family and fools,” and equity bank loan, public support, “friends, family and fools,” investors such as angel, seed or VCs, IPOs, etc. etc. Product • In search of a unique and profitable business • Often successful business models and numerous model, often enabled by or related to advanced products technologies Nature of support • Support in the nascent stages and critical first years • Support for mainly established businesses • Focus more on founders’ capabilities • Focus on firm managerial capabilities over founders Kind of support • Main support from specialized enablers such as • Main support from non-specialized enablers such as instruments accelerators, incubators, etc. enterprise/export development agencies, traditional BDS providers Results timeline • Results more mid-/long-term • Results more immediate/mid-term Note: BDS = business development services; IPOs = initial public offerings; SMEs = small and medium enterprises; VCs = venture capitalists. On the cusp of the new European programming period, Romania has a major opportunity to recalibrate its approach to innovation and entrepreneurship policy. The EU’s Recovery and Resilience Facility makes some €29 billion available in loans and grants to support reforms and investments in Romania to mitigate the economic and Executive Summary 13 social impact of the COVID-19 pandemic, while €44 billion will finance the Operational Programs under the 2021 programming period. €6 billion of the amount allocated through the Recovery and Resilience Plan are destined to the digital transition while €3.7 billion will finance the Operational Program Smart Growth, Digitalization, and Financial Instruments. The Recovery and Resilience Plan allocated financial resources to modernize the research sector and support the private sector in adopting digital technologies, but gaps seem to have remained — particu- larly in providing an instrument tailored to the specific needs of innovative firms and startups. Insights from the “Starting Up: Entrepreneurship Ecosystems in Romania” Report According to the European Innovation Scoreboard, Romania is a laggard in terms of its innovation and entrepre- neurship performance. Romania’s innovation performance is also a critical factor when measuring the strength of its entrepreneurship ecosystem. On some measures, the gap between Romania’s performance and the rest of the European Union (EU) has widened over the past decade. This review is therefore intended to support Romania in developing and implementing a set of policies that improve the country’s innovation and entrepreneurship ecosystem. Romania performs poorly on STI vis-a-vis its European peers. Innovation is a key driver of growth and productiv- ity. However, according to the EC’s European Innovation Scoreboard (EIS) 2021 —an index intended to measure the overall performance of EU Member States’ innovation systems—Romania has the weakest innovation system within the EU (figure 0.1). Performance actually declined between 2012 and 2020 relative to the rest of the EU: the innovation gap between Romania and other countries widened.  FIGURE 0.1  European Innovation Scoreboard Index 2021 160 140 120 100 80 60 40 20 0 Romania Bulgaria Latvia Poland Slovakia Hungary Croatia Greece Portugal Lithuania Czechis Spain Slovenia Malta Cyprus Italy EU-27 Ireland France Estonia Austria Luxembourg Germany Netherlands Belgium Denmark Finland Sweden 2014 2020 2021 Source: European Innovation Scoreboard 2021. Note: Colored columns show countries’ performance in 2021, using the most recent data for 32 indicators, relative to that of the EU in 2014. The horizontal hyphens show performance in 2020, using the next most recent data, relative to that of the EU in 2014. Grey columns show countries’ performance in 2014 relative to that of the EU in 2014. For all years, the same measurement methodology has been used. The dashed lines show the threshold values between the performance groups, where the threshold values of 70 percent, 100 percent, and 125 percent have been adjusted upward to reflect the performance increase of the EU between 2014 and 2021. EU = European Union. EU-27 = The 27 EU member states excluding the United Kingdom. Romania performs poorly across the board on innovation. Examining specific components of the European Innovation Scoreboard (European Commission 2021), Romania ranked last for Human Resources (a compos- ite measure based on education), second-last for Attractive Research Systems (a composite measure based on scientific publications), third-from-last for Digitalization (a measure based on the level of digital technolo- gies and includes two indicators, broadband penetration among enterprises and the supply of individuals with above basic overall digital skills; this dimension replaces the European Innovation Scoreboard 2021 dimension Executive Summary 14 on Innovation-Friendly Environment), fifth-from-last for Finance and Support (a measure based on public sector research and development (R&D)), last for Firm Investments (a measure based on business sector R&D), last for Innovators (a composite measure based on numbers of SMEs innovating), last for Linkages4 (a composite meas- ure based on collaborative R&D and publication), last for Intellectual Assets (a composite measure based on intellectual property registration), last for Employment Impacts5 (a composite measure based on employment in knowledge intensive or fast-growing enterprises), and eleventh-from-last for Sales Impact6 (a composite measure based on knowledge intensive exports). Romania also performs poorly on entrepreneurship. The European Index of Digital Entrepreneurship Systems (EIDES) provides a more specific composite measure of Romania’s entrepreneurship ecosystem. This index con- siders factors relating to culture and informal institutions, formal institutions, regulation and taxation, market con- ditions, physical infrastructure, human capital, knowledge creation and dissemination, finance, networking and support. EIDES ranks Romania 26th of 28 countries, although this index suggests that some improvement has been made between 2018 and 2020. Research also suggests that time is required to build the factors that con- tribute to entrepreneurial activity; for this reason, a country’s current position on the index may not reflect the recent changes in the ecosystem. Nevertheless, this index suggests a deficiency in Romania’s current entrepre- neurship policies, albeit with some improvement in conditions since 2018 (see figure 0.2 below). FIGURE 0.2  EIDES Scores Against Change in Score 2018 – 20 12 Ireland 10 France Finland Denmark Estonia Sweden 8 Romania EIDES 2020 score Spain Hungary Slovenia Poland Austria Netherlands 6 Lithuania Latvia Italy Bulgaria Croatia Portugal 4 Cyprus Czechia Belgium United Kingdom Greece Germany Luxembourg Malta Slovakia 2 20 30 40 50 60 70 80 Change in EIDES Score, 2018-2020 Note: EIDES = European Index of Digital Entrepreneurship Systems. Although firm entry levels are relatively high, the quality of Romanian firms at entry remains an issue, with Romanian firms are also less likely to innovate or scale up compared to their international peers, suggesting deficiencies in the entrepreneurship ecosystem. Entrepreneurship ecosystems comprise physical capital and infrastructure, human capital, access to finance/capital, regulations, support organizations, and access to mar- kets. Other critical dimensions include culture — including attitudes towards entrepreneurship, knowledge capital 4. Linkages includes three indicators measuring innovation capabilities: collaboration efforts between innovating firms, research collaboration between the private and public sector, and job-to-job mobility of human resources in science & technology. 5. Employment impacts measures the impact on employment and includes two indicators: employment in knowledge-intensive activities and employment in innovative enterprises. 6. Sales impacts measures the economic impact of innovation and includes three indicators: exports of medium and high-tech products, exports of knowledge-intensive services, and sales resulting from innovative products. Executive Summary 15 and social capital — which can impede or promote knowledge transfer between successful entrepreneurs and nascent startups. While firm entry itself is not a barrier, quality of firms at entry is a concern due to low levels of innovation and scaling up suggesting that there are barriers in the Romanian ecosystem that are preventing firms from achieving desirable outcomes. Skilled talent, whether as founders or workers in high-growth firms, contributes to high-quality firms at entry and scale up. Romania has a high proportion of science, technology, engineering, and mathematics (STEM) grad- uates, but this is not translating into a high share of R&D personnel in businesses, and many employers see the low-quality skills developed by the graduates to be a challenge. Although STEM training may be necessary for pro- ducing R&D personnel, data suggest that it is insufficient. One potential explanation for the gap may be “brain drain” (Gavriloaia 2020). In 2017, a significantly larger share of Romanians with a doctorate, masters, or college degree were living in a state different than the one in which they were born, compared to 2008 (Eurostat 2022). The differences amount to 144 percent between 2017 and 2008. Additionally, employment is a dominant motive among Romanian emigrants (OECD 2019b). While Romania has the fifth largest diaspora group in the world, and this is rapidly growing, this possibly signals continuous concerns about viable domestic employment opportunities. Improvement in entrepreneurship quality will also require more market opportunities for young and small businesses, both domestically and abroad. Barriers are preventing smaller firms from participating in public procurement. Public sector can create an important market for many startups and play a role in stimulating inno- vation. While recent reforms have improved awareness, current public procurement processes and platforms are still perceived as difficult to access for startups and other small firms. According to the most recent data avail- able (2019), SMEs bid on 35 percent of public procurement contracts but are awarded only 5 percent of them. Regarding access to external markets, Romania has fewer exporters than peer countries and uses a smaller pro- portion of imports in its exports. Only 17 percent of Romanian firms export more than 1 percent of their sales (compared to 29 percent of Slovakian firms). The use of e-commerce remains low, but uptake of e-commerce platforms has accelerated during the COVID-19 pandemic. Romania saw the share of firms selling online rise to 19 percent in 2020 from 9 percent in 2018. However, there remains vast unrealized potential: less than 20 per- cent of firms received even 1 percent of their revenue via online sales. Access to finance for startups and innovative firms remains a challenge. Romanian public sector financing mechanisms are not tailored for small firms and startups because they tend to favor post-revenue firms, which may have the effect of supporting incumbents over new entrants. Moreover, equity and debt financing for research and development remain below the EU average (European Commission 2017). Romania’s regulatory environment is not optimized for entrepreneurs and startups. This includes reforms to Romania’s company formation process, supporting distressed businesses, improving the ease of exiting a busi- ness, investing into startups, and intellectual property (IP) protections. This is because policymakers do not clearly distinguish between startups and SMEs and thus the former remain underserved by the existing regulatory envi- ronment (and policy mix). Though entrepreneurship is accorded reasonably high social status, increasingly considered a desirable career choice, and widely promoted by the media, Romania ranks low on indicators of social capital. Low social capital, which has been observed by the EU (European Commission 2017), is also evident in the ecosystem where stake- holders reported siloed networks and limited collaboration and connectedness between stakeholders, reinforcing the perception of ecosystem fragmentation. There also appears to be a limited “give back” mentality in Romania. “Intangible” entrepreneurial factors, such as risk appetite, attitudes towards entrepreneurship, and cultural mindset, are difficult to shift, though policymakers can ensure regulatory frameworks create an enabling envi- ronment for startups and entrepreneurs to start, scale up, and/or exit. Romanian authorities have introduced numerous policy instruments to support innovation and entrepreneurship. These ought to be revisited with the view to confirm to what extent they support the important issues identified in terms of entry and growth of spe- cifically high-growth firms. Part One analyzes the functionality of existing policy instruments. Part Two proposes additional policy recommendations to address deficiencies in the existing policy mix with a view of increasing the number of high quality and high-growth Romanian firms. Executive Summary 16 This report is divided into two main parts. Part One examines Romania’s Instruments for Entrepreneurship and contains analysis conducted by the World Bank team. It culminates with recommendations to improve the func- tionality of the Romania’s policy instruments. Part Two presents Policy Recommendation Notes for consideration as operational elements accompanying Romania’s National Startup Ecosystem Strategy. These could easily serve policymakers preparing Romania’s future science, technology, and innovation (STI) policy mix for the upcoming EU financing perspective. Most of the policy recommendations presented in this report overlap with the Top 12 Interventions identified by the Romania entrepreneurial ecosystem, revealing significant alignment between the ana- lytical findings and the demand for policy by key ecosystem stakeholders. Please see Annex A. Top 12 Interventions Identified by the Romanian Entrepreneurial Ecosystem for an overview of the Top 12 Interventions. At the request of the North-East Regional Development Agency (NE RDA), two additional Policy Recommendations are included within, “Scale up through exports” and “Implementing Startup Visas”. The team recommends additional analysis to inform a “Transform Public Procurement” recommendation that improves domestic market access for startups. Part One Part One provides an analysis of Romania’s STI policy instruments related to entrepreneurship. Innovation and entrepreneurship are key drivers of growth and productivity. An appropriate and effective portfolio of policy instruments relevant to innovation and entrepreneurship is thus important in driving economic growth, creating high-quality jobs, and increasing living standards, all of which are important objectives for Romania as it continues to advance its development. This analysis leverages the first two modules of the World Bank’s Public Expenditure Review (PER) framework (Correa 2014; Cirera and Maloney 2017). Part One is composed of two sections: an analysis of the current policy mix followed by a closer examination of the functioning of selected Romanian STI support programs. The first section (the “policy mix analysis”) examines the numbers and expenditures of the portfolio of instruments with regard to their objectives, types of intervention, benefi- ciaries, and so on. The second section (the “functional analysis”) assesses the functionality and governance of selected Romanian STI support programs. This exercise has its limitations because it was based on incomplete program data, a limited number of interviews and secondary research undertaken between late-2020 and mid-2021, and had to make several assumptions. However, it follows closely the methodology of the first two stages of the PER and provides reason- able indications both on the relevance of the policy mix and program functionality. This review did not cover instruments managed by the European Investment Fund (EIF) or look in detail at the public funding of research, IP regulations or technology transfer. Thus, Part One indicates areas where additional data collection and analysis would be needed to confirm some of the initial findings on STI presented in this report. Nevertheless, the analysis clearly identifies several areas where there is scope for improvement. Particularly, it makes suggestions about reforms to the overall policy mix and gaps in support, as well as issues relating to the design, implementation, and overall governance of instruments. Key Findings: Policy Mix Analysis The current portfolio of policies is highly concentrated. Significant resources are devoted to a few instruments, and a ‘long tail’ of much smaller instruments that are likely to be operating below their optimum scale. Rationalizing or scaling some of these smaller instruments ought to be considered. Most instruments have multiple top-level objectives. For example, some instruments combine productivity improvement objectives with job creation objectives. This may suggest a lack of clarity in the design of instru- ments. Together with the large number of small instruments, this suggests that few are likely to target important objectives, and some are likely to be ineffective. There are gaps in intermediate-level objectives. We recommend the development of instruments that address management practices, linkages with foreign firms, and market access, as a means of achieving top-level objectives. There are important gaps in support for entrepreneurs and early-stage startups. There is little specialization among instruments focused on the private sector. We recommend the development of more targeted instruments for early-stage, pre-profit, startups, and individual entrepreneurs—as well as the intermediary organizations that Executive Summary 17 support such startups, such as incubators and accelerators. Some “entrepreneurship” instruments may reinforce the role of incumbents rather than encourage the growth of innovative startups. Most instruments have multiple intervention mechanisms. Different mechanisms typically require different capabilities that are not complementary and are rarely found in one agency or office at the same time. We recom- mend that instruments that invoke multiple mechanisms simultaneously be reviewed. Grants are the dominant intervention mechanism. This may be appropriate in a relatively under-developed inno- vation and entrepreneurship ecosystem. However, it is important to remember that not every element of the eco- system can necessarily be resolved through public money. In Romania missing complementary aspects likely merit inclusion of other kinds of financial instruments and technical assistance. Key Findings: Functional Analysis Logic models are consistently weak. The analysis found a persistent weakness in the logic models (or theories of change) of instruments. Developing and articulating these models should enable better accounting of inputs and activities by managing authorities (MAs), as well as improved targeting of the instrument. It should also prompt ideas for alternative instruments and mechanisms, which would assist with rationalization of the portfolio and the development of a wider range of mechanisms. Instruments’ budgets were often mis-sized. This suggests that some rationalization—or expansion of the more successful programs—may be warranted. Instrument-level evaluation needs to be improved. Evaluation is currently stronger at the Priority Axis level. Improved evaluation at the instrument level would also support rationalization of the portfolio by providing infor- mation about which instruments should be scaled up and which may need to be discontinued. Administrative burdens for many beneficiaries are still high. There has been a significant effort by managing bodies to simplify the application processes for support schemes and reduce the bureaucracy for applicants. However, there are still indications that these processes remain overly complex for applicants. For example, the fact that startups and SMEs often need to hire consultants and advisors in order to apply for government support schemes is an indicator that the application processes are overly complex. Notably, this may have the effect of shifting support away from the earliest stage resource-scarce firms, where it is most needed. Bureaucratic burdens for administrators are also significant. Auditing of instruments (multiple times) appears to impose a significant burden on administrative teams. Bureaucratic friction is likely to be increased by the struc- ture of many instruments, which have both an MA and a separate intermediate body (IB); this may be a necessary condition of some European funding but is unlikely to be optimal. Policy Recommendations Part Two presents evidence and stakeholder-driven policy recommendations separated into Notes for con- sideration as inputs into Romania’s National Startup Ecosystem Strategy. These Notes will form the basis of a comprehensive package of policies to be considered for implementation by the Romanian Authorities. While Part One presents recommendations on how to improve Romania’s current policy and instruments mix for STI with a special focus on entrepreneurship, Part Two presents details on the kinds of policies, programs, and institutions that would be needed to implement a high-growth entrepreneurship ecosystem. Many of the policy recommen- dations presented in this report overlap with the Top 12 Interventions identified by stakeholders of the Romania entrepreneurial ecosystem, revealing consistency between the analytical findings and the demand for policy by key actors. Upon final consultation with the NE RDA, two additional Policy Recommendations were included as Notes, “Scale up through exports” and “Implementing Startup Visas.” The recommended approach to implemen- tation draws upon a wider review of the literature on the impact of programs to support entrepreneurship, includ- ing the World Bank’s A Practitioner’s Guide to Innovation Policy (Cirera et al. 2020). Executive Summary 18 The policy recommendations are categorized by: • Policies — legislative reforms and regulations to create an enabling environment for entrepreneurship, par- ticularly high-growth firms to thrive. • Programs — financial and non-financial support that targets entrepreneurs, firms, and other ecosystem actors. • Institutions — the entities and rules that are critical to govern the implementation of the national entrepre- neurship agenda. The table 0.2 below illustrates the categorization. It is possible for policy recommendations to span two or three categories given their foundational and cross-cutting nature, as indicated by table 0.2 below. TABLE 0.2  Categorization of Policy Recommendations Refocus attention on starting and scaling high-quality innovative firms Improve governance & functionality of existing instruments Policies Programs Institutions • Recalibrate the policy mix for starting • Strengthen Ecosystem Enablers • Establish a one-stop agency or and scaling high quality innovative • Create a Startup Fund “ecosystem hub” firms and improve the functionality of • Build and Promote a Network of instruments Romanian Founders & Diaspora • Reform Regulations to Strengthen • Implement Startup Visa Program Entrepreneurship & Investments • Scale-Up through Exports • Improve Entrepreneurship Education • Incentivize innovation to foster knowledge spillovers into the private sector • Promote the Digital Economy Source: World Bank Group. Part Two contains elaborations on the following 11 policy recommendations Policies 1. Recalibrate the policy mix for starting and scaling high quality innovative firms by (a) Improving the function- ality of instruments, and (b) Implementing a comprehensive package of reforms tailored to high quality inno- vative firms (that is, the National Startup Ecosystem Strategy). 2. Reform regulations to strengthen entrepreneurship and investments refers to initiatives that ease starting and exiting a business; incentivizes appropriate sources of financing into startups; and addresses Intellectual Property (IP) protections. 3. Implement Startup Visas refers to immigration incentives to attract skilled talent and investors. Programs 4. Strengthen ecosystem enablers refers to a pilot program to build capacity and deepen networks and link- ages of Romanian ecosystem actors. 5. Create a Startup Fund refers to establishing a fund that directly invests into riskier stage firms. 6. Build and promote a network of Romanian founders and diaspora refers to leveraging exposure and exper- tise of successful founders and diaspora to advise on critical issues such as market access and resources. Executive Summary 19 7. Scale-up through exports refers to initiatives to help Romanian startups access international markets. Institutions 8. Establish a one-stop agency or “ecosystem hub” refers to the formation of a centralized institution to imple- ment programs and policies identified under the National Startup Ecosystem Strategy and the Startup Fund. Policies, Programs, and Institutions 9. Improve entrepreneurship education and strengthen the role of universities in the ecosystem refers to human capital related measures to improve the quality of Romanian startups. 10. Incentivize innovation to foster knowledge spillovers into the private sector refers to ensuring startups and firms can benefit from R&D infrastructure. 11. Promote the Digital Economy identifies three subcomponents for consideration by Romanian policymakers. They are (i) promoting e-commerce platforms; (ii) increasing digital skills; and (iii) improving managerial capa- bilities to enable technology adoption. These recommendations also identify prioritization (or sequencing), time sensitivity, and “quick wins”7 (see ta- ble 0.3.). Our codification reflects the following: • Mission critical refers to activities that are (i) extremely time sensitive because the government is currently designing the new programming period, which provides an opportunity to embed critical policy recommen- dations; and (ii) lay the groundwork for the implementation of complementary Flagship recommendations in the future. If these recommendations are not immediately prioritized and programmed, Romanian Authorities risk not supporting these for another programming cycle. • Flagship refers to critical programs that should be undertaken to further development of Romania’s emerg- ing entrepreneurship ecosystem. • Foundational long-term refers to essential complementary activities, that are not exclusively in the domain of entrepreneurship policy, and that require a longer-term horizon to bear fruit. TABLE 0.3  Prioritization of policy Recommendations Policy Recommendations Prioritization Time Sensitive Quick Win Recalibrate the policy mix for starting and scaling high quality innovative firms by (a) Improving the functionality of instruments, and (b) Implementing Mission critical Yes Yes a comprehensive package of reforms tailored to high quality innovative firms       Reform regulations to strengthen entrepreneurship & investments Mission critical Yes Yes       Establish a one-stop agency or “ecosystem hub” Mission critical Yes Yes       Strengthen ecosystem enablers Flagship Yes     Create a startup fund Flagship   7. This refers to whether it is visible, has immediate benefit, and can be delivered quickly. Executive Summary 20 Policy Recommendations Prioritization Time Sensitive Quick Win Improve Entrepreneurship Education and strengthen the role of Universities in Flagship Yes the ecosystem     Implement Startup Visa Program Flagship Yes Yes       Build and promote a network of Romanian founders and diaspora Flagship   Scale-up through exports Flagship   Foundational Incentivize innovation to foster knowledge spillovers into the private sector   Yes longterm   Foundational Promote the digital economy   Yes longterm   Figure 0.3 illustrates the overlap between evidence- and stakeholder-driven policy recommendations. Our anal- ysis and interventions identified through the bottom-up strategy development process identified several similar policies, programs, and institutions. The major value addition of the data-driven diagnostic stems from the analy- sis of public instruments supporting STI and entrepreneurship, which identified an urgent need to recalibrate the policy mix and improve its functionality. the Romanian ecosystem also identified the need to appoint chief tech- nology officers (CTOs) in government. While we concur with this recommendation, the authorities need to prior- itize several other foundational interventions for the CTO recommendation to be fruitful. FIGURE 0.3  Overlap between Evidence- & Stakeholder-Driven Policy Recommendations Diagnostic- & Stakeholder-Driven Policy Recommendations in this Report Diagnostic-driven Stakeholder-driven Policy recommendations (co-created) policy policy identified in both approaches recommendations recommendations Recalibrate policy mix to Reform regulations to Implement a startup visa Appoint CTOs in include support for riskier strengthen entrepreneurship program Government4 stages of entrepreneurship & investments Build and promote a Transform public network of Romanian procurement1 founders & diaspora2 Establish a One-Stop Scale-up through exports Agency “Ecosystem Hub” Incentivize Innovation to Strengthen ecosystem promote knowledge enablers spillovers to private sector Create a Startup Fund Share R&D infrastructure3 Improve entrepreneurship Promote the digital education economy Notes: 1 This recommendation is incorporated into the “Reform regulations to strengthen entrepreneurship & investments” intervention 2 This recommendation overlaps with an activity identified under the “Scale-Up through Exports” stakeholder-driven intervention 3 This recommendation is incorporated into the “Incentivize Innovation to promote knowledge spillovers to private sector” intervention 4 We concur that this is a critical recommendation but there are many other foundational aspects that need to be prioritized given the foundational nature of this activity Executive Summary 21 Estimated total costs for all proposed 11 interventions. (See table 0.4.) The breakdown is as follows: TABLE 0.4  Estimated Costs for Proposed Policy Recommendations Activity Estimated Cos (€)t 1 Recalibrate the policy mix for starting and scaling high quality innovative firms by (a) 15 million Improving the functionality of instruments, and (b) Implementing a comprehensive package of reforms tailored to high quality innovative firms 2 Reform regulations to strengthen entrepreneurship & investments 2.6 million 3 Establish a one-stop agency (or “ecosystem hub”) 12.05 million 4 Strengthen ecosystem enablers 14.5 million 5 Create a startup fund 61 – 111 million 6 Improve Entrepreneurship Education and strengthen the role of Universities in the ecosystem 38.5 million 7 Implement a Startup Visa program 750,000 8 Build and promote a network of Romanian founders and diaspora 2.5 million 9 Scale-up through exports 7.325 million 10 Incentivize innovation to foster knowledge spillovers into the private sector 246 million 11 Promote the Digital Economy 150 – 300 million Total 550.225 – 750.225 million 22 PART ONE  AN ANALYSIS OF ROMANIA’S POLICY MIX AND FUNCTIONALITY OF ITS ENTREPRENEURSHIP PROGRAMS 23 CHAPTER 1  OVERVIEW  Chapter 1 Overview 24 Comprehensive and targeted policy can help stimulate entrepreneurial activity, giving rise to economic growth, innovation and job creation. An appropriately tailored portfolio of policy instruments can encourage and support the needs of a given entrepreneurship ecosystem. However, poorly designed public instruments can displace or inhibit private initiative and innovation. It is therefore important to undertake periodic reviews of the policy mix, spe- cific instruments, its targets, and recalibrate this based on the evolving needs of the entrepreneurship ecosystem. Part One of this report reviews Romania’s instrument set in support of entrepreneurship. Entrepreneurship support is a subset of the broader set of science, technology, and innovation (STI) instruments. The report follows the first two modules of the World Bank’s Public Expenditure Review (PER) methodology, which is aimed at deter- mining the quality of government measures to enhance the competitiveness of the country’s economy. The full PER in STI consists of four main components. They are (1) STI portfolio mapping and analysis of the pol- icy mix; (2) functional and governance analysis; (3) efficiency analysis; and (4) effectiveness analysis. The mod- ular methodology was refined in several analogous PER exercises in other European Union (EU) countries and beyond, including Bulgaria, Croatia, and Poland. It was calibrated to support EU authorities throughout the lifecy- cle of financial perspectives, from planning through mid-term program calibrations to post-programming impact evaluations. When fully implemented, the STI PER documents the quality of the state’s interventions in STI and its ability to enable the achievement of the social, economic and cultural objectives of the nation. The complete PER methodology with all its modules is illustrated in figure 1.1 (Correa 2014; Cirera and Maloney 2017). FIGURE 1.1  Schematic of the PER Stages Possible technical support Evaluates the extent to which policy outputs are being Evaluates the composition of transformed into expected Evaluates the quality of administrative costs vs the design, implementation and outcomes value of benefits transferred, governance of programs across programs across three dimensions—31 Evaluates alignment of budget categories allocation with national development objectives ROI/effectiveness Efficiency analysis Functional analysis Portfolio mapping and governance and analysis analysis of the policy mix Source: World Bank Group Note: PER = Public Expenditure Review; ROI = return on investment. Part One, Analysis of Romania’s Existing Policy Mix and Functionality, includes the first two modules of the four-stage PER: (1) the policy mapping and policy mix analysis, and (2) the functional and governance analysis. Due to data limitations, the conclusions offered within are preliminary, yet indicative of reality. Some conclu- sions are substantial enough to suggest potential policy improvements in the areas of entrepreneurship, innova- tion, and research and development (R&D). However, owing both to limited data availability and access to public program managers, most findings need more in-depth investigation to fully ascertain the public policy conse- quences of what is observed. This consideration is essential to keep in mind when interpreting the conclusions presented at the end of this portion of the report. Chapter 1 Overview 25 Romania’s science, technology, innovation, and entrepreneurship system performs poorly, with significant opportunity for stronger public policy intervention. Innovation is a key driver of growth and productivity. As pre- sented in the Introduction, Romania has the weakest innovation system within the EU, as measured by the indi- cators of the European Innovation Scoreboard (EIS) (European Commission 2021). In fact, performance declined between 2012 and 2020 relative to the rest of the EU: the innovation gap between Romania and other countries widened. It follows that there is significant room for improvement in Romania’s current STI policies. On the current trajectory, the existing policies will be insufficient to close the innovation gap between Romania and other EU states. 1.1 DEFINITION OF “INSTRUMENT” The PER STI methodology distinguishes public policy “instruments” by their causal mechanisms. The PER focuses on specific public interventions in STI. The definition of a public policy “instrument” is based on the causal ​​ mechanism that the intervention invokes to produce the desired changes in the given area of interest. The defini- tion of instrument used here may differ from the nomenclature of the Romanian government. The definitions used in the regulations serve to identify budget categories that allow a clear reference to the fiscal order. The definition used here is based on the need for the PER methodology to associate budget categories with effects on society and the economy. Doing so requires distinguishing instruments by means of the causal mechanism by which the desired results are expected to occur. Scholarships for postgraduate studies abroad are one example. In order to increase the population of highly qualified people with world-class experience, the state offers scholarships for postgraduate studies abroad for national students. The intervention, a subsidy in the form of a scholarship in this case, allows students to cover the high costs of their studies. In return, the country receives the benefits of the contributions of these graduates when they return. In this case, the mechanism for granting scholarships with the transfer of resources involved is an instrument of the state’s STI policy. Its implementation operates under the assumption that the country will not have the human resources in the necessary quantity and quality for its development unless it intervenes with the scholarships, and that these students will return to the country following their studies. 26 CHAPTER 2  POLICY MIX ANALYSIS  Chapter 2  Policy Mix Analysis 27 2.1 PURPOSE This section covers the first module of the PER of STI policy in Romania. It includes policy measures in the areas of entrepreneurship, as well as areas of innovation and research and development (R&D) that are poten- tially relevant for entrepreneurship. This also allows policymakers to understand entrepreneurship with the back- drop of the broader STI policy mix. The budget analysis examines the relationship between expenditures and objectives. In the budget analysis of the policy mix, the focus is on the relation of the expenditures in each instrument and the entire set of instru- ments (that is, the “policy mix”) with the patterns of objectives, types of intervention, beneficiaries and the fea- tures of firms, in the case of instruments aimed at the private sector. This analysis provides a baseline for the interpretation of further stages of the PER in STI policy. 2.2 METHOD The first step was to compile a list of relevant instruments and programs involving public resources. This list explicitly included sub-national (regional), national, and supra-national (EC) sources, provided that the instrument was at least partially administered within Romania and the recipients themselves were within Romania. This long list included several programs from the Programul Operațional Competitivitate (POC), a 2014-20 program sup- ported by the European Regional Development Fund (ERDF) and intended to increase competitiveness and eco- nomic development by improving information and communication technologies (ICT), research, and innovation; the Programul Operațional Regional (POR), a regional development program in Romania, also supported by the ERDF, whose purpose is the development of infrastructure and business environment; the Programul Operațional Capital Uman (POCU); the European Program for the Competitiveness of Small and Medium Enterprises (COSME); and the European Investment Bank (EIB). This stage did not, however, include block funding to universities and research organizations for basic research, nor did it include all European Investment Fund (EIF) funding. Programs were then filtered for relevance. Having compiled the list of programs, these were filtered for their rel- evance for entrepreneurship. Only programs with direct or indirect relevance to entrepreneurship were included. Thus, programs that supported private sector digitalization were retained, for instance, because some of these pro- grams might benefit digital startups. However, programs relating only to digitalization of government were excluded. Budget data was extracted from available documents and estimates are likely incomplete. This data was based on publicly available documents concerning the relevant instruments, as well as mySMIS, a common data portal for many of the European-funded programs. The data set was not complete, with some programs having no identifiable budget data. It is therefore important to note several caveats: first, data might be skewed by external reporting factors. Second, where programs had no budget data identifiable, estimates were made based on past programs or other sources, which might be inaccurate. Third, some programs covered a wide range of activities including entrepreneurship; in such cases, the propor- tion of spending that was relevant was estimated based on numbers reported by the managing authorities (MAs) at the oper- ational program level. Fourth, some instruments targeted multiple analytical categories such as multiple firm sizes or mul- tiple objectives; where this was the case, the total budget of the instrument was divided equally between the components. Budgets were standardized over time. The years of implementation for instruments were standardized assum- ing that the starting year was the year of the first call, and that the end year was the projected end year (with EU funds running to 2023, as per regulations, unless otherwise stated). The total budget of the instrument was then divided evenly by the number of years from the start year until the projected end year. The resulting averages were multiplied by the number of years passed since launch and summed up in the case of multiple calls. Although we recognize that the real spending profile of many programs is unlikely to be spread uniformly in this way, this aver- age allows an approximate comparison of budgets for programs of different lengths. We also recognize that, in the case of instruments with multiple calls, there is an issue regarding overlapping calls; the methodology may lead us incorrectly to assume that a greater proportion of the budget is spent in years where calls overlap. Conclusions should be considered carefully. Given the various caveats about the data and the process, the level of accuracy is not 100 percent. Therefore, analysis and recommendations made within this report should be treated with caution, even though they tend to mirror reality. Chapter 2  Policy Mix Analysis 28 2.3 FINDINGS 2.3.1 BUDGET The overall policy mix comprises some 50 instruments with a budget allocation over €3.7 billion. These fig- ures relate to the total budget allocation, including European funds (but excluding EIF investments) over the period under consideration. The available data for sources of funding shows that EU funding versus national funding is received in approximately a ratio of 3:2. This includes national co-financing and private sector funds that may be leveraged by public funding, although this ratio should be interpreted cautiously because there are missing offi- cial values for several national instruments. FIGURE 2.1  Distribution of Estimated STI Public Budget Allocation in Romania by Top-level Objective, 2014–20, in € Million and % Entrepreneurship R&D 25% (13 instruments) (8 instruments) € 948.98 m € 408.52 m 11% Digitalization / R&D Other (2 instruments) 2% (8 instruments) € 129.36 m 3% € 86.11 m Digitalization (7 instruments) 8% € 293.87 m Innovation & Competitiveness 4% (10 instruments) Digitalization / Innovation & Competitiveness € 1719.70 m (2 instruments) 46% € 148.25 m Note: Policy mapping of 50 instruments (2014–20), of which 13 were affiliated with entrepreneurship, 11 were affiliated with digitalization, 12 with innovation and competitiveness, and 10 with R&D. Primarily European Structural and Investment Funds (ESIF) and national funds. A little over one-quarter of the policy mix has entrepreneurship as its core objective. Thirteen of the 50 instru- ments, with a related total budget allocation of €949 million, have entrepreneurship as an explicit goal. The remain- ing three-quarters of the policy mix relate primarily to innovation & competitiveness, R&D, or digitalization. These, however, may be considered indirectly relevant to entrepreneurship or the development of the broader ecosystem. Standardizing STI budgets over time provides an estimated annual budget allocation of €577 million. As dis- cussed above, instruments vary in duration. If one adjusts for this, dividing each instrument’s allocated budget over the instrument’s years of operation and then summing these, the average total annual budget amounts to €577 million. (Alternatively, if one presumes that all spend will occur within the current seven-year funding period, as used for EU instruments, then the annual expenditure may instead be estimated at around €534 million.) This assumes that the budget of each instrument is distributed equally over their years of operation. It also assumes that all instruments are operating concurrently, which is representative of the most recent period under investi- gation (2019–20), but not necessarily for earlier years. Low absorption means that this is likely an overestimation of actual expenditures. The fact that a budget has been allocated does not mean that the amounts were actually spent or that the projects consummated. It is com- mon knowledge that Romania has very low absorption rates—that is, the proportion of budgets that are actually spent in time. This occurs for several reasons, including a lack of suitable institutional capacity at various levels. Data from October 2020 indicates that this absorption rate was only 34.7 percent for programs administered by the Human Capital Program (POCU), 31.8 percent for the Regional Operational Programme (POR), and as low as 26.5 percent for the Competitiveness Operational Programme (POC). In what follows, we have typically referred to the allocated budget because this better reflects governmental priorities. However, it should be noted that the actual expenditure on innovation and entrepreneurship are likely much lower due to the low absorption rate. Chapter 2  Policy Mix Analysis 29 There is a high concentration of resources in a few instruments. The graph below illustrates the cumulative annualized budget, sorted from largest to smallest instrument. It shows a significant concentration of allocated financial resources in a few large instruments. The three largest instruments—one under the Programul Ajutoare de stat pentru finanţarea proiectelor pentru investiţii and two under the Programul Operațional Regional (POR)— account for half of the annualized budget. Three-quarters of the total annualized expenditure is directed towards roughly a quarter of the total number of instruments analyzed in this exercise. FIGURE 2.2  Cumulative Annualized Budget by Number of Instruments 100 90 80 Cumulative Annual Budget (%) 75% of the budget 70 60 50 40 30 20 10 0 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 Number of Instruments Source: World Bank Group A corollary of the high concentration of resources in a few instruments is that there are many small instru- ments that may not have the scale necessary to have an impact even if they are well designed and imple- mented. Although it is not unusual to find such a distribution of budgets — a similar pattern is found in other coun- tries — it is important to recognize that, at the individual instrument level, the smaller instruments may represent a disproportionately large administrative effort. More than half the instruments in this set involve less than €5 million average annual expenditures. 2.3.2 OBJECTIVES In terms of general objectives, the profile of the policy mix seems focused on developing the business eco- system. Twenty-one (21) instruments focus on productivity improvements, 18 on new ventures and diversification, and 17 on skills and jobs. Knowledge creation, however, is represented by less than half the number of instru- ments focusing on other objectives. Furthermore, there FIGURE 2.3  Instruments by General (Top-Level) are no entrepreneurship instruments focusing explicitly Objective on the role of startups in addressing environmental or climate change objectives. Productivity Diversification & new ventures Most instruments focus on multiple general objec- Human capital tives. Thirty of the 50 instruments in this set are focused Societal on two general objectives simultaneously. The instru- Knowledge ments focused on social development and human cap- creation ital objectives are the ones that most often share their Environment aim with another objective (Fig. 2.4). In other words, 0 5 10 15 20 25 there are very few instruments that focus solely on these Instruments (no.) objectives exclusively. On the other hand, about half the Source: World Bank Group instruments that focus on productivity, do so exclusively. Note: Instruments can have multiple objectives. Chapter 2  Policy Mix Analysis 30 FIGURE 2.4  Number of General Objectives Supported by Instruments Productivity Diversification & new ventures Human capital Societal Knowledge creation Environment 0 2 4 6 8 10 12 14 16 18 20 22 Instruments (no.) Sole objective Dual objective Source: World Bank Group It is also possible to classify instruments according to more specific objectives, which may be considered as the intermediary route by which the general objective is to be achieved. For example, an instrument might have the ultimate goal of improving productivity, and approach this via improving management practices, or technol- ogy adoption, or something else. Note that although some of the specific objectives are likely to be more closely linked with some of the general objectives than others, they are not necessarily subsets of the general objectives.) The most common specific objectives are technology transfer, skills formation, business environment improve- ment and entrepreneurship, but many instruments have multiple specific objectives (figure 2.5). Between 15 and 20 instruments include one of the top four as their specific objective. On the other hand, management prac- tices, linkages with foreign firms, and market access are addressed by very few instruments (fewer than 5 for any one of them). FIGURE 2.5  Instruments by Specific (Intermediary) Objective Skills formation Technology transfer & translation Entrepreneurship Improving business environment Access to finance Business R&D and related innovation Non-R&D innovation or tech diffusion Research excellence Market access (domestic) Foreign/domestic firm links Management practices Export promotion 0 5 10 15 20 Instruments (no.) Source: World Bank Group Note: Instruments may be counted more than once. Many instruments have multiple specific objectives, and several instruments have four of them. Although skills formation and technology transfer may seem to be the two most common objectives for instruments, none of the instruments are strictly dedicated to these objectives. The lion’s share of the resource allocations (around three-fifths of the total budget) goes to two general objec- tives: productivity growth and diversification/new ventures. The other objectives reflect a small fraction of the annualized expenditure. Thus, the distribution of expenditures by general objective reveals a significantly differ- ent picture from the one produced by the number of instruments per objective. About as many instruments are dedicated to societal development and human capital as to productivity growth and diversification/new ventures; however, the budget dedicated to the instruments focused on societal development and human capital is much smaller than the budget dedicated to the instruments that focus on productivity and diversification. Chapter 2  Policy Mix Analysis 31 FIGURE 2.6  Instruments by Specific Objective (Sole versus Mixed) Skills formation Technology & translation Entrepreneurship Improving business environment Access to finance Business R&D and related innovation Non-R&D and related innovation Research excellence Market access (domestic) Foreign/domestic firm links Management practices Export promotion 0 5 10 15 20 Instruments (no.) Sole objective Two objectives Three objectives Four objectives Source: World Bank Group FIGURE 2.7  Annual Budget by General Objective Productivity Diversification & new ventures Human capital Societal Knowledge creation Environment 0 20 40 60 80 100 120 140 160 180 200 220 240 Annual budget (€, millions) Source: World Bank Group Note: Where instruments have multiple objectives, the budget is split evenly across them. Two specific objectives receive large shares of the budget expenditures: (a) non-R&D innovation with technol- ogy adoption and diffusion and (b) access to finance (figure 2.8). Technology transfer, improving the business environment, and entrepreneurship follow at about half the size of expenditures of the larger ones. Other specific objectives receive significantly smaller budget allocations. Thus, the distribution of expenditures by specific objec- tives shows a significantly different pattern from the distribution of the number of instruments by specific objectives. FIGURE 2.8  Annual Budget by Specific Objective Skills formation Technology & translation Entrepreneurship Improving business environment Access to finance Business R&D and related innovation Non-R&D and related innovation Research excellence Market access (domestic) Foreign/domestic firm links Management practices Export promotion 0 20 40 60 80 100 120 140 Annual budget (€, millions) Source: World Bank Group Note: Where instruments have multiple objectives, the budget is split evenly across them. Chapter 2  Policy Mix Analysis 32 2.3.3 TYPES OF INTERVENTIONS Grants (including matching grants) are the preferred method of intervention. Forty-three of the 50 instruments are in this category (figure 2.9). The type of intervention represented by each instrument is an important feature of public policy, and the combination of types of inter- vention reflects the multiple causal mechanisms that the government’s strategy invokes to obtain the de- FIGURE 2.9  Number of Instruments by Type of sired STI results: if the distribution by objectives indi- Intervention cates what public authorities want to achieve, the dis- Grants tribution by types of intervention indicates how they are Business networking acting. (See appendix B for a more complete descrip- Science & technology parks tion of these classifications.) Technology extension R&D infrastructure Entrepreneurship education Several instruments use more than one intervention Incubators & accelerators mechanism simultaneously. In some cases, the use of Business services multiple intervention mechanisms may be associated Public procurement with the fact that several instruments are centers or or- Tax incentives Open innovation ganizations that contain multiple types of intervention Loan subsidies within them. However, this is not the case for all instru- Credit guarantees ments; other instruments are not centers or organiza- Equity finance tions but nevertheless have multiple intervention mecha- Vouchers nisms. This pattern of multiple intervention mechanisms 0 5 10 15 20 25 30 35 40 45 per instrument is consistent with that observed in oth- Instruments (no.) er less developed innovation systems. Source: World Bank Group For example, the largest instrument invokes three different intervention mechanisms: vouchers, tax incentives, and infrastructure. These are not easy to manage together given the different assumptions in each. Vouchers are instruments to purchase R&D services performed by another entity selected by the beneficiary. Tax incentives apply to the beneficiaries for their own performance of R&D. Infrastructure financing is a supply side instrument for pub- lic goods (research facilities in higher education institutions, for example). This combination is highly unusual. An- other example is the fifth largest program in the portfolio—2.1.B Incubatoare de afaceri (POR 5)—which provides not only grants but also business services, early-stage infra- structure, collaborative networks and education for en- trepreneurship. In this case, because the purpose of the FIGURE 2.10  Estimated Annual Budget by Type of instrument is to set up incubators, the funding to set it Intervention up and the incubators’ activities are included as specific Grants objectives. This case is less problematic, but the report- Business networking ing confuses the level at which the intervention operates. Science & technology parks The only valid mechanism in this case is the grant. Net- Technology extension working and training, for example, are not carried out by R&D infrastructure a government agency. It could be the case that the grant Entrepreneurship education Incubators & accelerators to set up incubators is provided in combination with train- Business services ing, networking and business services by a government Public procurement agency. In that case, the intervention would be a portfo- Tax incentives lio and would have to be designed as such. The evidence Open innovation does not suggest this is the case for this intervention. Loan subsidies Credit guarantees Equity finance Grants also dominate the budget. The distribution of Vouchers resources (Figure 2.10) again confirms the dominant 0 40 80 120 160 200 240 role of grants, with science & technology parks, busi- Annual budget (€, millions) ness collaboration networks, vouchers, and tax incen- tives leading in importance. Source: World Bank Group Chapter 2  Policy Mix Analysis 33 2.3.4 TYPES OF BENEFICIARIES Half of the instruments benefit firms. Universities or research institutes are next, followed by business consor- tia, clusters, or associations (figure 2.11). This is consistent with the prominence of productivity and diversifica- tion as key objectives of STI policy. FIGURE 2.11  Number of Instruments by Type of Direct Beneficiary Firms Research institutes & universities Consortia, associations, clusters Government entities Entrepreneurs Researchers Business enablers Financial Institutions 0 2 4 6 8 10 12 14 16 18 20 22 24 26 Instruments (no.) Source: World Bank Group Firms are also the main recipients of funding, but individual researchers follow closely. (Figure 2.12) The budget emphasis on individual researchers was not reflected in the number of instruments. This means the three instru- ments focused on individual researchers as beneficiaries have, on average, larger budgets than the 25 instru- ments focused on firms. Business consortia and associations still appear prominently in budget expenditures, but universities and research institutes receive a smaller proportion of resources given the number of instruments. (It should be noted that there are other funding instruments for universities that were not examined in this PER because they did not relate to entrepreneurship.) It is also very notable that business enablers — including accel- erators and incubators — appear to have both a relatively low number of instruments and a relatively small overall budget directed towards them, and individual entrepreneurs have very little direct funding. FIGURE 2.12  Estimated Annual Budget by Beneficiary Firms Research institutes & universities Consortia, associations, clusters Government entities Entrepreneurs Researchers Business enablers Financial Institutions 0 10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 160 170 180 190 Estimated annual budget (€, millions) Source: World Bank Group 2.3.5 FIRM CHARACTERISTICS Instruments focused on firms primarily target micro, small, and medium enterprises (MSMEs). The policy instru- ments that indicate firms among their foci (that is, most of the instruments) are uniformly distributed across MSME firm sizes. Large firms are targeted by fewer than half as many instruments as those targeting the other firm sizes (figure 2.13). The distribution of disbursed funds by projects to various beneficiaries might reveal that certain types of firms received more support than others. However, information on beneficiaries of disbursed funds and patterns of supported projects was not available. It was only possible to determine the documented aim of the instruments. SThe allocation of yearly average budgets follows the same pattern. (Figure 2.14). Both the number of instru- ments and the budget allocations show that many instruments target several firm sizes simultaneously. In fact, 35 of the 50 instruments target at least 3 of the 4 different firm sizes. Chapter 2  Policy Mix Analysis 34 FIGURE 2.13  Instruments by Firm Size FIGURE 2.14  Estimated Annual Budget by Firm Size 40 150 Estimated annual budget (€, millions) 35 125 30 Instruments (no.) 100 25 20 75 15 50 10 25 5 0 0 Micro Small Medium Large Micro Small Medium Large Source: World Bank Group Source: World Bank Group Note: Instruments may be counted more than once. Micro = 0 – 9 Note: Micro = 0 – 9 employees; Small = 10 – 49 employees; Medium = employees; Small = 10 – 49 employees; Medium = 50 – 249 employees; 50 – 249 employees; Large = 250 or more employees Large = 250 or more employees Few instruments target firms at the seed/pre-seed phase. If one looks at firm maturity instead of size, a slightly different picture emerges. The distribution of the number of instruments by firm life cycle shows that the seed/pre- seed phase has fewer than half of the number of instruments of the other three phases on average (figure 2.15). Mature firms receive more monitoring and evaluation (M&E) budget allocation than early-stage firms. Twice as much funding is available to mature firms as to firms in the other stages of development (Figures 2.15 and 2.16). The other three stages, which could be interpreted as higher-risk propositions, receive roughly half of the mature firms’ average yearly budget. FIGURE 2.15  Instruments by Firm Maturity FIGURE 2.16  Yearly Average Budget by Stage of Firm Maturity 26 24 500 Average yearly budget (€, millions) 22 450 20 18 400 Instruments (no.) 16 350 14 300 12 250 10 200 8 150 6 4 100 2 50 0 0 Seed Startup Scaleup Mature Seed Startup Scaleup Mature Source: World Bank Group Source: World Bank Group 35 CHAPTER 3  FUNCTIONAL ANALYSIS Chapter 3  Functional Analysis 36 3.1 PURPOSE This section covers the second stage of the PER of STI policy in Romania. It aims to assess the functionality and governance of selected Romanian STI support programs, thus providing evidence-based recommendations for improving the design, implementation, and governance of the country’s STI portfolio. The analysis assesses the functionality, rather than the impact, of STI support instruments; functionality, here, refers to the quality of processes involved in creating and implementing each instrument, including the design, implementation, M&E systems, human resources, and governance (the instrument’s integration and interactions with other programs, institutions, and regulations). 3.2 METHOD The methodology for the analysis is based on a comprehensive analytical framework benchmarked to interna- tional best practices. The analytical framework, developed by the World Bank, scores support programs along a total of 30 variables: 14 related to program design, 12 related to implementation, and 4 related to program gov- ernance (figure 3.1). See Annex B. Instruments Examined for the overview of examined instruments. See Annex C. Detailed Findings of the Functional Analysis for more detailed findings from the Functional Analysis, including the approach for the data classification. The functionality of each program is scored on a scale of 1 to 5 for each variable, where 5 denotes international best practice. A score of 3 should not necessarily be interpreted as ‘aver- age’ but as a functional aspect that is satisfactory but has room for improvement. FIGURE 3.1  Summary of Functional Analysis Indicators DESIGN IMPLEMENTATION GOVERNANCE Origin Learning Relationship between instruments Justification Solicitations Relationship between institutions Relationship with portfolio Eligibility criteria Relationship with other policy frameworks—awareness and Objectives Application and selection process adjustment Choice of instrument Information management Relationship with other policy frameworks—severity of limitations and modifiability Logic model Project closures and follow-up Inputs Budget management and organizational quality Activities Roles and autonomy Outputs Staff and training Beneficiaries Incentives Selection criteria Process monitoring Audiences Monitoring & evaluation Results and impact implementation Monitoring & evaluation design Source: World Bank Group Chapter 3  Functional Analysis 37 The analysis aims to reveal whether key elements of effective innovation policies have been met in three key areas: • Design. Public interventions must be designed through a process covered by the rule of law and consist- ent with general national or regional goals for research and innovation. They must be properly justified and address real problems, avoiding the trap of addressing false failures. This justification can also help avoid capture of public resources by certain vested beneficiaries. Once a system failure is identified, policy mak- ers should consider the full range of alternative intervention designs rather than simply copying existing pro- grams. By design, interventions should follow a clear, well-articulated logic model that helps depict the shared relationships and causal linkages between program inputs, activities, outputs, and outcomes and logically connect them to higher-level strategic objectives. Logic models should define indicators for inputs, activities, outputs, and outcomes that allow for M&E of program performance. • Implementation. Processes for administering the program, including application, selection, and reporting, should be clear and transparent, and knowledge management systems should be in place to allow for sys- tematic learning and improvement of the instrument during implementation. Implementing agencies must have adequate human resources and organizational structures to administer the program, and staff should have training opportunities and incentives that are relevant to program performance (rather than generic to public administration staff). Internal and external M&E of the instrument should take place, and, critically, evaluation results should be used to improve and adapt the program. • Governance. Coordination mechanisms should be in place to minimize overlap and enhance complementa- rities between the instrument and other programs and agencies. Implementing staff should also be aware of external laws and regulations that can inhibit the implementation of the instrument and should be proac- tive in taking steps to adapt for optimal operation of the instrument. This analysis covers nine instruments supporting STI covering the period 2014 to 2023. Together, these nine instruments represent around €1.4 billion of the €3.7 billion of the total allocated funding (around 37 percent). However, many of the remaining set of 50 instruments fall under the same operational programs and ministries, and so will share some similar characteristics. The key agencies and ministries interviewed (which included the Ministry of EU Funds and Investments; the Ministry of Public Works, Development and Public Administration; and the Ministry of Finance) were responsible for the administration of the bulk of the instruments in the policy mix (instruments totaling €3.2 billion of the €3.7 billion), and in many cases confirmed that other instruments were designed and operated in similar fashion to those ones under analysis. Thus, although the functional analysis only examined nine instruments, there is reason to believe that this sample reflects the wider policy mix. Nevertheless, the observations and conclusions from this analysis should be viewed as indicative of possible issues but not nec- essarily conclusive, as the small sample may not capture many potential areas of interest regarding practices that affect the functionality of the overall set of STI policies. The analysis was based on semi-structured interviews. Interviews were held with key staff from the instruments concerned, each scored by a team composed of a minimum of three interviewers, in order to try to minimize inter- viewer bias. On occasion, multiple interviews were held for one instrument, either because administration of an instrument was split across organizations, or because clarification was needed. It is important to note that the National Authorities were not the primary client of this project and agreed to participate at the request of regional authorities. As a result, the project did not have the internal motivation to connect with the project’s potential as an opportunity for learning and continuous improvement. Therefore, the interviews were not as informative and lacked the detailed narratives about organizational culture and tacit knowledge that was made available for the functional analysis in other countries. Some information was gleaned from reports by stakeholders rather than the program managers themselves. However, the perception of outsiders cannot substitute for direct reports from the managers themselves. Chapter 3  Functional Analysis 38 3.3 OVERVIEW OF FINDINGS Romania’s scores show some areas that are ripe for improvement. Some scores are higher than others. In addi- tion, there is significant variation within indicators when considering the average scores in comparison to the max- imum and minimum score for each indicator across the set of instruments included in the analysis. We discuss design, implementation, governance, and issues related to variation. Several of the areas show influence from EU regulations. In general, EU regulations appear to have had a positive impact on functionality, including in the program origins, closure, M&E, and coordination mechanisms. However, we also noted areas where unclear or inadequate EU rules may have contributed to low scores, particularly in the use of logic models and the consideration of alternative instruments. We suggest that it would help Romanian authorities to differentiate more clearly between compliance and functionality: these are not the same, and sev- eral instruments appeared to demonstrate a focus on the former at the expense of the latter. 3.3.1 DESIGN Among the design indicators, those related to the use of logic models, accounting of inputs and activities, and the proper identification and measurement of outcomes deserve special attention. The areas of specif- ic opportunity for improvement are reflected in the indicators with lower scores in figure 3.2. The problem areas have average scores below 3, the middle of the range or point of indifference. The high average score for program origin is also noticeable. The remaining indicators have average scores slightly above 3, reflecting better practic- es but with room for improvement. The existence and application of a logic model and the proper cataloging of inputs and activities are two areas of concern. The FIGURE 3.2  Average Scores for Design indicators lower scores of these indicators reveal a possible dis- Program origin connect between the destination of the expenditures of M&E the instrument and the administrative costs incurred to Audiences implement it given that cataloging of inputs and activi- Portfolio relationship ties relates directly to the ability to estimate such costs. Products/outputs The quality of the definition of objectives of the instru- Selection criteria Category ment and the determination of outcomes and impacts Main beneficiaries are other areas of concern. For example, objectives are Program justification Program objectives often self-referential statements to the need for the in- Activities strument. In other words, in the case of funding instru- Alternative instruments ments, the objective of an instrument is to provide fund- Inputs ing in a certain area because there is little funding or Expected outcomes support in that area. This sort of statement of objec- Logic model tives is circular and begs the question of why the gov- 0 1 2 3 4 5 ernment must support it and what outcomes are expect- Score (5 is best) ed from providing it. Source: World Bank Group The program origin has good practices in general, which is explained by the processes required by the opera- tional programmes (OPs) of the EU framework. These processes are highly formalized requiring documentation and good rationales for inclusion of instruments in their funding schemes. This avoids pitfalls related to arbitrary reasons, imitation, or undue influences. However, it is possible that this origin has also affected deeper consider- ation of some other design features; more than once, interviewees commented that design of an instrument had been determined or heavily influenced by the European Commission (EC) rather than having been designed by the Romanian authority itself following a consideration of various alternatives. The relatively high average score for inclusion of an M&E system deserves further scrutiny. The score for this indi- cator is mainly due to the existence of a centralized data gathering and reporting system, which offers good infra- structure for data and information management when it comes to M&E. However, it is well known from other coun- tries that such centralized systems are often tailored to general government investments and expenditures and so do not include specific items that are tailored to science, technology and innovation policies. Chapter 3  Functional Analysis 39 Moreover, the fact that both objectives and outcomes are not as good raises the question of how the sys- tem operates in relation to the instrument implementation. Most of the responses received from the interviews revealed that program managers did not distinguish the objectives, inputs, activities, outputs, and outcomes of projects from the policy or instrument, and it appeared that M&E was often weaker at the instrument level than at the program level or Priority Axis level. Supported projects do not automatically aggregate to successful gov- ernment interventions, and so the outcomes of these instruments may not be good enough to influence the sys- tem, even if each individual project within it has good results. Therefore, the design of the M&E system may not be what the instrument really needs. We therefore recommend that this aspect of functionality is examined in closer detail in the future. 3.3.2 IMPLEMENTATION Among the implementation indicators, eligibility and selection criteria, budgets, and incentives associated with staff performance are of concern. The implementation indicators related to eligibility, budget, staff perfor- mance incentives, and process monitoring have lower average scores. Scores on these indicators fall below the midrange of the scale (as shown in figure 3.3 below). The application process reflects the side of the beneficiar- ies in relation to submitting proposals and expecting a decision in a timely manner. There is some variety across instruments, but the division of labor in process- FIGURE 3.3  Average Scores for Implementation ing applications with different roles given to different indicators entities (financial and technical assessments, for exam- ple) and some repetition in verification processes limit Project closure Program database timely responses. Solicitations Applications The budget indicator average score is relatively low. Roles This reflects the difficulties that many instruments have Learning Category in executing it, and not having a good understanding of M&E the demand for the specific focus in the design of the Program mgmt. solicitations. There were problems with both undershoot Staff and training and overshoot, suggesting that instrument budgets are Process monitoring not designed commensurate with instrument goals. Eligibility criteria Incentives Romanian STI instruments are not hiring specialized Budget/finance human resources, connecting performance assess- 0 1 2 3 4 5 ment with program goals, or monitoring for continuous Score (5 is best) improvement. Most of the reporting on management Source: World Bank Group and staffing indicated a degree of satisfaction with the Note: M&E = monitoring and evaluation. structure and availability of human resources. However, the scores may overestimate the actual situation because it was common to hear during interviews that the public service system operated under general rules, and the managers did not think much of the relation between those and the specifics of the STI policies under study. There is consensus in developed nations and the analytical liter- ature on management of public support for R&D and innovation that specialized human resources are necessary with flexible entrepreneurial approaches to public management. In this vein, the performance assessment of staff was not related to the content of the instruments. This disconnect between performance assessment and program goals occurs in other areas and is the general established process of public service. Similarly, the processes of public management were not monitored for continuous improvement, which is reflected in its lower average score. Implementation capacity requires further examination. We suspect that implementing capacity is lacking in Romania as it is in neighboring countries for which functional analyses were also conducted. Reasons to suspect that this may be a constraint include the fact that some instruments did not seem to have enough skilled staff nor many opportunities for improving their own staff. This is reinforced by evidence from other World bank projects (such as the “Supporting Innovation in Romania’s Catching-Up Regions” project) as well as from ecosystem stakeholders. Regrettably, we were unable to examine this in detail as our research did not allow direct observation of the working conditions, nor was the Romanian Government the ultimate client. However, we suggest that it warrants closer attention. Chapter 3  Functional Analysis 40 Many instruments were structured in a way that is likely to increase administrative costs. It is notable that many instruments were structured in a way that involved both an MA and an intermediate body (IB) working in coordination. This was generally reported as working satisfactorily, with clearly defined responsibilities. However, from a management theory perspective, as well as from practical experience of similar situations in other coun- tries, it seems unavoidable that this structure will introduce some additional friction and costs. For example, in case adjustments are needed during instrument implementation or impediments arise due to application of reg- ulations or EC rules, this arrangement might call for consultations and lengthy negotiations. Situations such as these were found in all other countries in the region where a functional analysis was conducted (Bulgaria, Croatia and Poland). A follow up of this analysis may be necessary to assess the proper operation of this division of labor to understand whether the functionality of any instruments is affected. 3.3.3 GOVERNANCE The analysis found relatively good internal and external coordination mechanisms (coordination with programs within the agency and with other agencies), as shown in figure 3.4 below. This is partly because programs within the EC’s ‘priority axes’ have some coordination mech- anisms built in. However, in some cases those mecha- nisms could be utilized further to provide more strategic FIGURE 3.4  Average Score for Governance perspectives on the related programs. Indicators The functional analysis also found several instanc- Program relationship Category es of entrepreneurship programs being constrained Jurisdictions (internal) by European State Aid rules. This was primarily due to Institutions relationship the State Aid rules not being well-adjusted to the inno- Jurisdictions (seriousness) vation support by government interventions. However, these rules are not easy to change, and managers were 0 1 2 3 4 5 Score (5 is best) typically aware of these rules and their constraints but cannot change or mitigate their effect. Source: World Bank Group 3.3.4 EVIDENCE FROM VARIATION There is significant variation in the scores across indicators for this set of instruments, indicating opportu- nities for learning within Romania. Variation refers to how well policy instruments scored against the best-case scenario, according to global practice. Higher scores are coded in green, average scores are coded in yellow, and lower scores are coded in red. The combined radar graphs in figure 3.5 illustrate variance of public instruments. This chart shows the maximum and minimum scores for each indicator in addition to the average score for each indicator. Some indicators have relatively high scores in at least one instrument, even when the average is quite low. Conversely, several indicators have the lowest possible score as well. This indicates that there is potential for internal learning from the examples of better practices in some of the instruments. Comparing scores and variability reveals system wide strengths, needs for system wide improvements, and opportunities for learning from within. Another view of the variation of scores across programs is presented in figure 3.6 below, which plots the average score per indicator on the horizontal axis and the standard devia- tion on the vertical axis. indicators that fall in the upper two quadrants (those with high standard deviations) are areas within the set of policies with a high potential for internal learning, as there are programs in these areas with both high scores (good practices) and low scores (poor practices). Variables that fall in the upper left quad- rant show great variation in scores but lower averages, so learning opportunities may be the greatest in these areas. Variables in the bottom left quadrant are those with both low scores and low standard deviation, indi- cating the need for systemwide improvement with few, if any, examples of good practices in this set of policies. Indicators in the bottom right quadrant are those with high scores and low standard deviation, indicating areas of systemwide strength. Chapter 3  Functional Analysis 41 FIGURE 3.5  Functional Analysis Results — Scores by Category across the Set Interaction of Regulations (External) Program Origin Program Justification Interaction of Regulations (Internal) Portfolio Relationship Institutions Relationship CE Program Objectives AN ERN 5 V Programs Relationship GO Alternative Instruments 4 Monitoring, Evaluation and Learning Logic Model 3 Process Monitoring Inputs DES 2 IGN Incentives 1 Activities 0 Staff and Training Products/Outputs Roles and Autonomy Main Beneficiaries IMP LEM Program Mgmt Selection Criteria /Organization Quality EN TA TIO Budget and Financial Resources Audiences N Project Closures Expected Outcomes and Impact Program Database /info on Participants and Applications Monitoring and Evaluation Application and Selection Processes Learning and Adjustments Eligibility Criteria Solicitations /Application Information Minimum scores Average scores Maximum scores Source: World Bank Group FIGURE 3.6  Average Score and Standard Deviation Across All Instruments 1.4 Staff/Training Institutions Relationship Roles/Autonomy Jurisdiction (Internal) 1.2 Solicitations Alt. Alternative Insttument Instruments Programs Relationship Org. QualityQuality Organization Portfolio Relationship Standard deviation Elegibility Eligibility Info. Information 1.0 Process Monitoring Program Database Jurisdiction (External) Activities M&E (1) Audiences Justification 0.8 Selection Processes Objectives M&E(D) Inputs Beneficiaries Products Outcomes Incentives 0.6 Sel. criteria Selection criteria Logic Model Budget Learning 0.4 1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 Average score Source: World Bank Group Note: M&E = monitoring and evaluation. Chapter 3  Functional Analysis 42 There are several indicators that seem to need systematic improvement. These indicators (in the lower left quadrant) are design indicators associated with the use of logic models, inputs and activities of the instrument rather than supported projects, and the matter of setting instrument objectives and definition of specific out- comes and impacts. There is also an implementation indicator in this group, namely, the budget, that seems also to be a systemic issue. There are significant opportunities for learning, including from internal capabilities. The indicators in the upper left quadrant where there are significant opportunities for learning, including from internal capabilities, include the consideration of an alternative instrument for the same objective on the design side, the eligibility criteria and application information and process monitoring on the implementation side, and the effect of rules from other jurisdictions. Mixed indicators and strengths. There are many indicators in the upper right quadrant that have averages above the midpoint but also rather high variance, indicating that there are cases with poorer practices in those areas as well. Finally, there are a few indicators that show good practices systematically, shown in the lower right quadrant. These are the origin of instruments and their closure practices. These two indicators benefit from OP requirements that establish clear rules for these areas of design and implementation. Accounting for products of the instrument, identification of beneficiaries and selection criteria in design, and ability to learn systematically from experience with the instrument are other areas of better practice. There is some more variation on the M&E by design that is in this quadrant by a small margin, which is related to how well adjusted the centralized digital platform used for M&E is to the needs of specific instruments. See Annex C. Detailed Findings of the Functional Analysis for more detailed findings from the Functional Analysis. 43 CHAPTER 4  CONCLUSIONS Chapter 4 Conclusions 44 The analysis above leads to various conclusions about the policy mix and function of the instruments that Romania is using to promote entrepreneurship. The first conclusion that may be drawn is that the concentration of spending on a few instruments leaves many smaller instruments that may be ineffective. Within the complete collection of STI instruments at the national level (the “policy mix”), spending is highly concentrated on a few instruments. The largest instrument amounts to about half the yearly average budget, while three of the 50 instruments account for half the total average yearly budget. This high concentration means that the remainder is composed of many instruments with small budgets that are probably not of sufficient scale to have an impact, given that some of the administration costs are likely to be fixed. Second, the issue with the small scale of many instruments is compounded by the fact that many of them aim for several objectives simultaneously. Along with the high concentration of the budget on a few instruments, many instruments attempt to achieve a diversity of objectives (both general objectives and specific objectives), and there is overlap between them. The combination of a relatively large number of instruments with small budgets and several objectives suggests that there may be lack of clarity in the design of instruments to focus the intervention. Most instruments already have a relatively small scale, and the problem is compounded by the lack of specialized focus, because more resources are required to attend to multiple objectives. Having many interventions with small budgets and complex purposes raises an important question about the potential effectiveness of the policy mix. Third, most instruments assume that money is the only barrier to achieving changes in the innovation sys- tem. This assumption can be inferred from the fact that most instruments use grants or matching grants as the mechanism of intervention. The dominance of grant-based subsidies is not unusual. It is present in many coun- tries. However, it must be kept in mind that grants assume that the only problem that needs resolution from the public intervention perspective is lack of money and that all other factors contributing to innovation, entrepre- neurship, productivity and competitiveness are available or reachable if money is made available. This assump- tion is almost certainly false: it is highly unlikely that this is the true condition of the STI system in Romania or any other nascent innovation system. Moreover, grants can sometimes have negative effects. The potential negative effects can include market dis- tortion (for example, disadvantaging other businesses that do not receive grants, thereby distorting investment and other economic activity), grant dependency (for example, creating organizations that cannot compete without grants or subsidies and so rely on continued public support) and lobbying (for example, increasing the incentive for firms to lobby the government instead of engaging in productive activities). Given the stage of development of Romania’s ecosystem, we suggest that such distortionary effects are not a concern at this time. Nevertheless, it is important to review the effectiveness of grants as agents of change in STI. (For example, recent work on technol- ogy adoption suggests that managerial practices and the lack of knowledge and skills within the firms is often a greater barrier to adoption than the access to the material technology itself, so grants solely for technology acqui- sition are likely to have limited impact.) Many instruments use multiple intervention mechanisms that are unlikely to be effective in all of their dimen- sions. Numerous instruments invoke multiple causal mechanisms to influence the system. In addition to the pre- vious observations about the small scale of a great majority of instruments and the diversity of their objectives, it is very difficult for them to be effective in all these dimensions. Given the expected complexity of multiple types of intervention, the administrative cost of these instruments is very likely to exceed their specific budget and sig- nificantly reduce their chances of impact. Different mechanisms typically require capabilities that are not comple- mentary and are rarely found in one agency or office at the same time. Therefore, instruments that have multiple forms of intervention may create challenges for efficient administration and should be reviewed. In addition to the diversity of objectives and types of intervention, many instruments also indicate multiple types of beneficiaries. All the comments on the multiplicity of intervention mechanisms are applicable here. The best practices for managing public instruments suggest specialization in terms of beneficiaries, even within the same class. This finding is another reason to recommend a rationalization of STI instruments. The lack of specialization among instruments focused on the private sector may further challenge their effec- tiveness. There is little specialization among instruments focused on the private sector. Many of them simultaneously Chapter 4 Conclusions 45 target firms of several sizes and life cycle stages. However, firms of different sizes and stages of the life cycle face very different challenges and have different needs. Experience indicates that each of these business activity seg- ments requires special skills to serve them (For example, the policies to support a small startup with high-growth ambition, potentially working in a field of emerging technology, may be very different from the policies needed to support SMEs in general.) It is very difficult for a uniform set of interventions to effectively address all these is- sues simultaneously. Small instruments with such an ambitious spectrum of care requirements are highly unlike- ly to have such capabilities. It is worth considering fine-tuning these instruments to address a better characteri- zation of firm needs by targeting their intervention. In addition, where targeting exists, this may sometimes be counterproductive to the creation of new high- growth ventures. For example, several instruments are targeted at mature firms (regardless of their size) than for firms in the very early stages of starting up; this may potentially reinforce the role of incumbents rather than encouraging the growth of innovative startups. In addition, several instruments are available only to firms that are already profitable; however, experience from other countries suggests that many startups that are high growth (and ultimately very profitable) may undergo a considerable pre-revenue period, during which they are focused on growth and user-adoption. Relatedly, many instruments specifically target established technologies at high tech- nology readiness levels (TRLs), that is, mature, market-ready technologies;however, a supportive innovation eco- system needs to provide support for technologies across the spectrum of technology readiness, from the labora- tory to the market. Very few instruments address management practices, linkages with foreign firms, or market access. Academic research indicated that these play an important role in technology adoption and exporting respectively, but these areas receive little attention. Moreover, although skills formation and technology transfer seem to be the two most common objectives for instruments, none of the instruments examined by this study were strictly dedicated to these objectives. Business enablers and individual entrepreneurs have little direct funding. In many ecosystems, business ena- blers — including accelerators and incubators — play an important role. However, such organizations appear to have both a relatively low number of instruments and a relatively small overall budget directed towards them. Individual entrepreneurs (as opposed to firms) also have very little direct funding. There are consistent weaknesses in the logic model of instruments. The functional analysis of the instruments showed that logic models were generally weak, as were the models’ associated inputs, objectives, outcomes and incentives. Weaknesses in the logic model of instruments, as well as mediocre justification of the programs, may possibly be linked with the origins of some instruments. In some cases, the starting point for the development of an instrument appears not to have been the internal identification of a market failure and the subsequent develop- ment of a logic model, but rather a desire to follow the direction of the EC and the Priority Axis objectives. However, it is important fully to understand the rationale and logic of the Commission’s guidance, and to translate this into a logic model for each instrument; a clearer logic model and justification will then support numerous other aspects of each instrument’s development, such as more specific objectives and outcomes, better accounting of inputs, improved targeting, and consideration of alternative instruments. Instruments’ budgets were often mis-sized. This includes both undershoot and overshoot. This issue is likely related to the problem of having many interventions with small budgets. This again suggests that some rationali- zation — or expansion of the more successful programs — may be warranted. Greater evaluation at the instrument level is needed. M&E was often weaker at the instrument level than at the program level or Priority Axis level. There appeared to be good processes for evaluating the overall impact of the portfolio of instruments, and also relatively good processes for tracking and evaluating individual projects that are supported by a specific instrument. However, evaluation and comparison of the instruments themselves seems to be less clear. Improving this will allow better decisions about how to rationalize the policy mix. The administrative structure of several instruments appears sub-optimal. Many instruments were structured with an MA and an IB. This seems very likely to introduce some additional friction and costs. Although this struc- ture may be a necessary condition for some European funds, it is important to pay close attention to the interface Chapter 4 Conclusions 46 between organizations. Improved process monitoring may help assess this friction and determine whether it is improving or worsening over time. In general, the dual role of EC rules and regulations was perceived in Romania as well as it neighbors that also receive EU structural funds. On the positive side, origination of policies, rules for closures, engagement with stakeholders are constructive elements of following EC guidance to set up the instru- ments along the axes of the EC OP in the country. On the other hand, in some areas compliance with the rules is substituted for better functionality because more technically sophisticated logic models and indicators useful for guiding M&E and implementation generally are not developed beyond minimum compliance requirements. The project did not have sufficient information on similarities and/or differences with nationally funded instruments to carry out a comparison. Implementation capacity requires further examination. There were suggestions from the research, supported both by ecosystem stakeholders and other World Bank research, that implementation capacity might be a signif- icant constraint. 47 CHAPTER 5  RECOMMENDATIONS TO IMPROVE POLICY MIX AND FUNCTIONALITY OF ENTREPRENEURSHIP PROGRAMS Chapter 5  Recommendations to Improve Policy Mix and Functionality 48 The following presents recommendations to improve the functionality of the existing policy mix with an indicative timeframe. Short-term refers to activities that can be accomplished within 18 months – 2 years and medium term refers to activities that can be accomplished within 2 – 5 years. Many of these activities are catego- rized as short-term due to the confluence of the programming period. The timing of this exercise coincides with Romania’s review and development of new operational programming. Lessons from this analysis can be applied to the future policy mix, especially when it comes to improving program management, which impacts how pro- grams are designed, implemented, and evaluated. #1: Rationalize the overall portfolio of instruments, to reduce the number of small instruments that are likely to be ineffective due to their scale. (Short-term) #2: Improve instrument-level evaluation, to enable better comparison of instruments’ effectiveness, and hence inform ongoing decisions about which instruments to scale up, with attention to smaller pilot instruments that may hold promise. Such evaluations should look at both efficiency and effectiveness, following modules 3 and 4 of the PER STI methodology. (Short-term) #3: Clarify the instrument’s objective (that is, the rationale or specific market failure it is trying to address) and consider instruments that are more focused and specific. Although multi-purpose instruments can be effective (and rationalization may push towards fewer instruments that are more general in scope), the current portfolio of multi-purpose instruments is likely to have a diluted impact. It may also make instrument-level evaluation and comparison more difficult. (Short-term) #4: Ensure that in the comprehensive set of instruments, there are specialized ones designed for companies in the early stages of technology readiness and market readiness, and for companies with aspirations for scale. There needs to be more support targeted towards pre-revenue startups and individual entrepreneurs. This includes addressing management practices, linkages with foreign firms and market access. This also includes separate, dedicated instruments for skills formation and technology transfer, rather than these being joint objectives along- side others. (Short-term) #5: Instruments should have an explicit logic model — a hypothesized chain of causes and effects leading to the desired outcomes. Further development of such models is linked with clearer justification for each instrument, and should enable better accounting of inputs, improved targeting of the instrument, as well as prompting ideas for alternative instruments. (Short-term) #6: Further support private-sector business enablers, including accelerators and incubators. In most ecosystems, these have an important role to play, although in Romania such organizations currently have a relatively small overall budget directed towards them. (Short-term) #7: Consider a wider range of mechanisms besides grants, tailored appropriately to type of firms and stage of firms. Grants are useful financing mechanisms for the earliest stages of entrepreneurs, including startups and firms with riskier business models, who often rely on self-financing. For other types of firms, loan guarantees, tax incentives, and reforms to public procurement may be as effective as grants in some circumstances, while hav- ing a less distortionary effect on markets and reducing the risk of grant-dependency. (Short-to-medium term) #8: Continue to reduce the administrative burden on applicants. Many instruments showed good signs of having been simplified or otherwise reformed to make it easier for applicants. However, the fact that many applicants still rely on third-party consultants and advisors indicates that there is room for further improvement. (Short-to- medium term) #9: Be aware of the administrative burden on staff. It was apparent from interviews that the burden of audits and other internal administration is a considerable cost in the management of many instruments. Particularly where instruments are sub-scale, this will reduce their impact. There may be opportunities to increase staff motivation. (Short-to-medium term) Chapter 5  Recommendations to Improve Policy Mix and Functionality 49 #10: Formalize learning and make better use of available data. Although there are clearly processes for the system- atic improvement of instruments, these could be further developed. In particular, there may be value in using the program database to analyze instruments further and understand how to make adjustments. This will be further enhanced when combined with logic models that have clear views of instruments’ inputs, processes, and outputs, and accounting of these resources (for example, the time taken at each stage). (Short-to-long term, continuous.) 50 PART TWO  EVIDENCE- AND STAKEHOLDER-DRIVEN POLICY RECOMMENDATIONS A REVAMPED & PRIORITIZED ENTREPRENEURSHIP POLICY MIX — NOTES ON POLICIES, PROGRAMS, AND INSTITUTIONAL INTERVENTIONS 51 CHAPTER 6  OVERVIEW OF POLICY RECOMMENDATIONS Chapter 6  Overview of Policy Recommendations 52 Part Two presents Policy Notes for the operationalization of Romania’s National Startup Ecosystem Strategy and its future entrepreneurship policy. While Part One presents recommendations on how to improve Romania’s policy mix for STI and entrepreneurship, this chapter takes a comprehensive approach by prioritizing key recommen- dations to improve the ecosystem for high-growth entrepreneurship in Romania. Many of the policy recommenda- tions presented in this report naturally overlap with the Top 12 Interventions identified by the Romania entrepre- neurial ecosystem, revealing consistency between the analytical findings and the demand for policy by key actors. See Annex A. Top 12 Interventions Identified by the Romanian Entrepreneurial Ecosystem for an overview of the Top 12 Interventions as identified by the Romanian entrepreneurship community. Upon final consultation with the North-East Regional Development Agency (NE RDA), additional stakeholder-driven recommendations are devel- oped in Part Two, “Scale up through exports” and “Implementing Startup Visas.” The recommended approach to implementation draws upon a wider review of the literature on the impact of programs to support entrepreneur- ship, including the World Bank’s A Practitioner’s Guide to Innovation Policy (Cirera et al. 2020). Each Policy Note follows a similar structure, including objective, target beneficiaries, alignment with Top 12 Interventions, rationale, approach, (possible) implementation body, risks, notional costing (amount)8, timeline, and key performance indicators (KPIs). The proposed timeline for implementation is categorized as short-term (within 2 years), medium-term (within 2–5 years); and long term (5 or more years). Where possible, notional costing is provided. With these considerations in mind, the recommendations are presented sequentially because many interventions are intertwined and depend on implementation of other critical elements. The policy recommendations are categorized by: • Policies — legislative/regulatory reforms to create an enabling environment for high-growth entrepreneurship to take root and thrive; • Programs — programs that target entrepreneurs, firms, and other ecosystem actors; and • Institutions —  governance and entities essential for the entrepreneurship agenda. The figure 6.1 illustrates the proposed sequencing and prioritization of the policy recommendations, which FIGURE 6.1  Sequencing of the Policy also take into account time sensitivity and “quick Recommendations wins”9. Our codification reflects the following: Timeline • Mission critical refers to activities that are (i) Mission Flagship extremely time sensitive because the government Critical is currently designing the new programming period, Recalibrate the policy mix Strengthen ecosystem Create a startup for starting and scaling enablers fund which provides an opportunity to embed data-driven high quality innovative and “SMART”10 policy recommendations based; Improve entrepreneur- Build and promote Reform regulations to ship education & a network of and/or (ii) lay the groundwork for future recommen- strengthen entrepreneur- strengthen the role of Romanian founders ship & investments universities and diaspora dations. If these recommendations are not imme- diately prioritized, Romanian Authorities risk miss- Establish a one-stop Implement startup visa Scale-up through agency “Ecosystem Hub” program exports ing its economic objectives. • Flagship refers to critical activities that should be undertaken to further development of Romania’s emerging entrepreneurship ecosystem. Foundational Longterm • Foundational long-term refers to critical activities Incentivize innovation to foster knowledge spillovers into the private sector that require a longer time horizon to come to fruition because there are other “foundational” elements Promote the digital economy that need to be sequenced and prioritized first. 8. This will require further calibration with input from the Authorities. 9. This refers to whether it is visible, has immediate benefit, and can be delivered quickly. 10. This refers to the following framework: Specific, Measurable, Attainable, Relevant, and Time-Bound. Chapter 6  Overview of Policy Recommendations 53 The table below illustrates the categorization. It is possible for policy recommendations to span two or three cate- gories, as indicated by table 6.1 below. The figure 6.2. below illustrates the overlap between evidence- and stake- holder-driven policy recommendations. Our analysis and interventions identified from the bottom-up strategy development process identified a number of similar policies, programs, and institutions. The main differences are based on an analysis of public instruments supporting STI and entrepreneurship, our analysis identified an urgent need to recalibrate the policy mix and improve its functionality; and the Romanian ecosystem identified a need to appoint chief technology officers (CTOs) in government. While we concur with this recommendation, the Authorities need to prioritize other critical interventions. TABLE 6.1  Categorization of Policy Recommendations Refocus attention on starting and scaling high-quality innovative firms Improve governance & functionality of existing instruments Policies Programs Institutions • Recalibrate the policy mix for starting • Strengthen Ecosystem Enablers • Establish a one-stop agency or and scaling high quality innovative • Create a Startup Fund “ecosystem hub” firms and improve the functionality of • Build and Promote a Network of instruments Romanian Founders & Diaspora • Reform Regulations to Strengthen • Implement Startup Visa Program Entrepreneurship & Investments • Scale-Up through Exports • Improve Entrepreneurship Education • Incentivize innovation to foster knowledge spillovers into the private sector • Promote the Digital Economy Source: World Bank Group. FIGURE 6.2  Categorization of Policy Recommendations Diagnostic- & Stakeholder-Driven Policy Recommendations in this Report Diagnostic-driven Stakeholder-driven Policy recommendations (co-created) policy policy identified in both approaches recommendations recommendations Recalibrate policy mix to Reform regulations to Implement a startup visa Appoint CTOs in include support for riskier strengthen entrepreneurship program government4 stages of entrepreneurship & investments Build and promote a Transform public network of Romanian procurement1 founders & diaspora2 Establish a one-stop Scale-up through exports agency or “ecosystem hub” Incentivize Innovation to Strengthen ecosystem promote knowledge enablers spillovers to private sector Create a Startup Fund Share R&D infrastructure3 Improve entrepreneurship Promote the digital education economy Note: 1This recommendation is incorporated into the “Reform regulations to strengthen entrepreneurship & investments” intervention 2This recommendation overlaps with an activity identified under the “Scale-Up through Exports” stakeholder-driven intervention 3This recommendation is incorporated into the “Incentivize Innovation to promote knowledge spillovers to private sector” intervention 4We concur that this is a critical recommendation but there are many other foundational aspects that need to be prioritized given the foundational nature of this activity Chapter 6  Overview of Policy Recommendations 54 Estimated total costs for all proposed 11 interventions. The breakdown is as follows: TABLE 6.2  Estimated Costs for Proposed Policy Recommendations Activity Estimated Cost (€) 1 Recalibrate the policy mix for starting and scaling high quality innovative firms by (a) 15 million Improving the functionality of instruments, and (b) Implementing a comprehensive package of reforms tailored to high quality innovative firms 2 Reform regulations to strengthen entrepreneurship & investments 2.6 million 3 Establish a one-stop agency (“or ecosystem hub”) 12.05 million 4 Strengthen ecosystem enablers 14.5 million 5 Create a startup fund 61 – 111 million 6 Improve Entrepreneurship Education and strengthen the role of Universities in the ecosystem 38.5 million 7 Implement a Startup Visa program 750,000 8 Build and promote a network of Romanian founders and diaspora 2.5 million 9 Scale-up through exports 7.325 million 10 Incentivize innovation to foster knowledge spillovers into the private sector 246 million 11 Promote the Digital Economy 150 – 300 million Total 550.225 – 750.225 million 55 POLICY RECOMMENDATION 1  RECALIBRATE THE POLICY MIX FOR STARTING AND SCALING HIGH QUALITY INNOVATIVE FIRMS BY (A) IMPROVING THE FUNCTIONALITY OF INSTRUMENTS, AND (B) IMPLEMENTING A COMPREHENSIVE PACKAGE OF REFORMS TAILORED TO HIGH QUALITY INNOVATIVE FIRMS zz MISSION TIME QUICK GOVERNMENT SUPPORT CRITICAL SENSITIVE WIN NEEDED      OBJECTIVE  Refocus attention on starting and scaling high-quality innovative firms       TARGET BENEFICIARIES  Young, innovative firms at the earliest stages of technology-readiness and market-readiness and eco- system enablers      RATIONALE  As discussed in Part One of this report, Romania’s current policy mix for STI and entrepreneurship is not optimized for firms at the earliest stages and entrepreneurs, nor does it address the quality of firms at entry and the limited number of high-growth firms in the ecosystem. Romania hovers at round 3 percent of high-growth firms compared to 11.4 percent in peer countries. This reflects policymak- ers’ limited understanding of how startups differ from small and medium enterprises (SMEs) because many instruments target post-revenue firms. Our functional analysis of existing support programs reveals that: • Many instruments are under-disbursing, suggesting misalignment between the instrument’s ob- jectives and target beneficiaries. • Several instruments are targeted at mature firms (regardless of their size) rather than for firms in the very early stages of development. • Additionally, several instruments are available only to firms that are already profitable. This likely excludes high-growth startups because they typically undergo a considerable pre-rev- enue period, during which they are focused on growth and user-adoption. During this stage, high-growth startups typically rely on grants or financing from accelerators, angel investors, or venture capitalists (VCs). • Relatedly, many instruments specifically target established technologies at high TRLs (that is, mature, market-ready technologies); however, a supportive innovation ecosystem needs to pro- vide support for technologies across the spectrum of technology-readiness. Policy Recommendation 1 56 Recalibrate the policy mix for starting and scaling high quality innovative firms • Individual entrepreneurs (as opposed to firms) have little direct funding. • Very few policy instruments benefit ecosystem enablers, such as incubators, accelerators, and other critical actors. Efforts to recalibrate the policy mix must also include improving attention to national and localized entrepreneurship ecosystems. As discussed in a forthcoming recommen- dation, Romania’s subnational entrepreneurial ecosystems vary in quality of support, which in turn, translates to limited support to startups and innovative firms. • Some instruments existed because of the direction of the EC, rather than having been identified as a particular systemic failure. • Administrative costs for program management are high because it requires coordination between an MA and an IB. • M&E capabilities of program staff are weak, limiting opportunities to understand the impact of existing instruments and scaling up of programs that are meeting and/or exceeding program ob- jectives. This indicates a need to improve capabilities of program implementors for design, im- plementation, and governance, with a special focus on M&E.      APPROACH  This recommendation is divided into three specific sub-categories: (1A) Rebalancing the policy mix, (1B) Improving functionality of the policy mix; and (2) Passing and implementing a comprehensive package of reforms, that is, the National Startup Eco- system Strategy. PART 1A: REBALANCING THE POLICY MIX As a first step towards rebalancing the policy mix, policymakers should rationalize the overall portfo- lio of instruments, to reduce the number of small instruments that are likely to be ineffective due to their scale with a view to reallocate these resources to launch new instruments targeting earlier-stage firms. There are many instruments with small budgets that are probably not of sufficient scale to have an impact, given that some of the administration costs are likely to be fixed. Policymakers should also clarify the instrument’s objectives and target beneficiaries. Many instruments list multiple objectives simultaneously and/or target multiple types of beneficiaries. It is very difficult for them to be effective in all these dimensions. Given the expected complexity of multiple types of in- terventions, the administrative cost of these instruments is very likely to exceed their specific budget and significantly reduce their chances of impact. PART 1B: IMPROVING FUNCTIONALITY OF THE POLICY MIX Romania currently has limited institutional capacity to design, implement, and evaluate instruments for firms at both levels. Relatedly, monitoring impact of programs is weak and there are missed op- portunities in identifying high-performing policy instruments and applying those implementation les- sons to other support programs. Policy Recommendation 1 57 Recalibrate the policy mix for starting and scaling high quality innovative firms Other activities to improve functionality of the policy mix include improving program governance, sim- plifying application processes to reduce administrative burden for potential beneficiaries as well as program staff; and improving the selection process. PART 2: PASSING AND IMPLEMENTING A COMPREHENSIVE PACKAGE OF REFORMS, THAT IS, THE NATIONAL STARTUP ECOSYSTEM STRATEGY Many countries are consolidating regulatory reforms under an entrepreneurship strategy, or under a Startup or Small Business Act. Such entrepreneurship strategies typically include specific policy sup- port and funding instruments for riskier firms in the earliest stages when the rationale for public inter- vention is greatest. Romania’s access to finance interventions embedded within the strategy should be informed by an in-depth study on the existing entrepreneurial finance. The startup ecosystem strat- egy also needs to take a nuanced approach towards supporting firms. While support to firms at the ideation stages remains a gap in public sector instruments, there also needs to be a focus on provid- ing support for startups throughout the entrepreneurial lifecycle. Romania’s National Startup Ecosystem Strategy should also include the establishment of a new in- stitution, a one-stop agency or “ecosystem hub,” that will be responsible for implementing programs and policies for entrepreneurs.       POLICY AND IMPLEMENTATION BODY  This agenda would benefit from commitment at the highest level of the Government. This is current- ly led by the Ministry of European Funds with support from the relevant line ministries including the Ministry of Entrepreneurship and Tourism; Ministry of Research, Innovation, and Digitalization (MOR- ID); and Ministry of Economy. At the implementation level, this is delegated to MAs of Ministries and Regional Development Agencies dealing with Regional Operational Programs.      RISK  A key risk for Recommendation 1, as well as Recommendation 2, is that this will require a high degree of internal coordination, including coordination with other strategies such as the Digitalization Strategy. The champion of these reforms will have to be able to work with authority and credibility across the mul- tiple departments and other bodies concerned; this will require strong leadership and political support. Other key risks include lack of awareness among policymakers about the specialized needs of start- up firms, which include a specialized ecosystem with funding, access to networks, mentors, and mar- kets. Rebalancing the policy mix (1A) and improving functionality of the policy mix (1B) also requires buy-in and coordination from the EC, MAs and the IBs Further risks include lack of a cohesive approach to supporting startups, such as a national start- up ecosystem strategy and lack of a centralized institution, such as a one-stop agency or “ecosystem hub” to lead implementation on this agenda. In many countries, innovation agencies oversee public in- terventions for STI and entrepreneurship, whereas startup support organizations engage in advocacy around the special needs of innovative growth-oriented firms. Romania lacks such institutional capacity.      AMOUNT  €15 million based on preliminary costing of calibrating policy mix plus capacity building to introduce functionality improvements on design, implementation, governance, and evaluation. Policy Recommendation 1 58 Recalibrate the policy mix for starting and scaling high quality innovative firms Additionally, estimated total costs for all policy recommendations are as follows: Activity Estimated Cost (€) 1 Recalibrate the policy mix for starting and scaling high quality innovative 15 million firms by (a) Improving the functionality of instruments, and (b) Implementing a comprehensive package of reforms tailored to high quality innovative firms 2 Reform regulations to strengthen entrepreneurship & investments 2.6 million 3 Establish a one-stop agency (or “ecosystem hub”) 12.05 million 4 Strengthen ecosystem enablers 14.5 million 5 Create a startup fund 61 – 111 million 6 Improve Entrepreneurship Education and strengthen the role of Universities in 38.5 million the ecosystem 7 Implement a Startup Visa program 750,000 8 Build and promote a network of Romanian founders and diaspora 2.5 million 9 Scale-up through exports 7.325 million 10 Incentivize innovation to foster knowledge spillovers into the private sector 246 million 11 Promote the Digital Economy 150 – 300 million Total 550.225 – 750.225 million      TIMELINE  Short-to-Medium-to-Long Term There is an immediate opportunity associated with the preparation of the new Operational Programs under the 2021 – 7 financial perspective. Once set in place, the policy mix will need to be continuous- ly revisited and recalibrated to meet the evolving needs of the sector, including at mid-term of the Op- erational Programming period. Long term activities will include capacity building, including in the ar- eas of governance and M&E.      KPIs  Outputs • Recalibrated mix of policies and instruments, resulting in a higher proportion of funding specifi- cally targeted at startups and the ecosystem • Passage of National Startup Ecosystem Strategy Outcomes • Improved public sector capacity for design, implementation & governance of entrepreneurship in- struments (reflected in improved scored in Functional Analysis assessment) • Improved functionality of public resources to startups and the ecosystem (reflected in improved scored in Functional Analysis assessment) • Increase in number of startups successfully accessing public support • Improved allocation of public resources to startups and the ecosystem 59 POLICY RECOMMENDATION 2  REFORM REGULATIONS TO STRENGTHEN ENTREPRENEURSHIP AND INVESTMENTS MISSION TIME QUICK CRITICAL SENSITIVE WIN      OBJECTIVE  Reform regulations to create an enabling entrepreneurship ecosystem. Specifically, reforms that dig- italize company formation process, scaling-up, receiving investments, improve ease of exiting a busi- ness, intellectual property (IP) rights, and accessing public procurement opportunities.      RATIONALE  Business environment is preventing firms from growing, scaling, and investments to occur      APPROACH AND TOTAL COSTS FOR COMPREHENSIVE PACKAGE OF REFORMS This recommendation is divided into four reform areas. The total costs for each reform area is listed in the table below. Activity Estimated Cost (€) Digitalize company formation (to promote ease of starting a business) 1 million Support distressed businesses and ease exits 400,000 Catalyze investments into startups 400,000 Clarify intellectual property rights 400,000 Improve access to public procurement opportunities 400,000 Total 2.6 million DIGITALIZE COMPANY FORMATION      RATIONALE  When it comes to ease of starting a business, Romania has met objectives established by the EC’s Directorate-General for Internal Market, Industry, Entrepreneurship and SMEs (DG GROW), “Start-up Procedures”. It takes 1 – 3 days to register a business and costs €26 (target is €100 or less). This is currently a paper-based procedure.      APPROACH  Digitalizing the company formation process can further reduce the number of steps and time to com- plete all procedures. By the end of 2022, the Government of Romania, through the national recovery Policy Recommendation 2 60 Reform regulations to strengthen entrepreneurship AND investments and resilience plan (NRRP), commits to adopt legislation to simplify and digitalize firm entry and exit in line with EU’s Digitalization Directive, remove redundant procedures, consolidates various licensing processes under a new one-stop-shop for industrial licenses and authorizations.       IMPLEMENTATION BODY  The implementation body for this reform is the Ministry of Entrepreneurship and Tourism in coordina- tion with the National Trade Registry Office and other agencies facing firms.      RISK  As with Recommendation 1, these tasks require significant internal co-ordination, a strong champi- on and political support. Lack of coordination between the key agencies is a key risk, as is ineffective management of transition, including software development, testing and conversion/migration, and lack of coordination with other Government projects aiming at digitalizing the public sector (for exam- ple, establishment of the Government Cloud, adoption and enforcement of the interoperability frame- work, insufficient training for staff in public agencies, low uptake among target groups).      AMOUNT  €1 million      TIMELINE  Short Term      KPIs  Output • Elimination of paper-based registration procedures Outcome • Number of companies registering online       ALIGNMENT WITH TOP 12 INTERVENTIONS  Intervention 1: Reform Startup & Investment Related Regulations SUPPORT DISTRESSED BUSINESSES AND EASE EXITS      RATIONALE  Although Romania performs well on entry of new firms, it does not have a high share of tech start- ups and especially high-potential startups. Much entrepreneurship in Romania is driven by necessi- ty, rather than opportunity, with many firms lacking the ambition to innovate or other characteristics to become high-growth firms. Additionally, Romania does not have a culture that destigmatizes “rapid business failure” and promotes serial entrepreneurship. Easing firm exit and providing entrepreneurs the possibility of discharge of debts and the possibility of a fresh start would help reduce the number of so called “zombie firms” and encourages the reallocation of resources to more productive new en- trant firms, while encouraging further entrepreneurship. Policy Recommendation 2 61 Reform regulations to strengthen entrepreneurship AND investments      APPROACH  This requires a major shift in mindset among policymakers, including by considering a special track optimized for high-risk business models and technology startups within existing distressed firm frame- works through early warning and restructuring tools, but especially exit. The lack of awareness of cur- rent insolvency legislation also appears to be a key barrier to accessing policy support, and barriers to receiving this support will need to be identified specifically for this set of firms. The EU Directive on Restructuring and Insolvency (Directive EU 1023/2019)11, which aims to align pre-insolvency proceed- ings across the EU Member States, highlights the need for all EU member countries (including Roma- nia) to open possibilities for a second chance for honest natural person entrepreneurs.       IMPLEMENTATION BODY  The implementation body for this reform is the Ministry of Entrepreneurship and Tourism with inputs from the Secretariat General of the Government, the Ministry of Economy, the Ministry of Justice, the National Trade Registry Office.      RISK  Policymakers need to understand that viability is a complex and nuanced concept that needs to ac- count for several factors, especially complicated during the pandemic. Influencing cultural mindset around firm failure is viewed as a less tangible activity.      AMOUNT  €400,000 to support the ongoing implementation of the Directive and provide capacity-building train- ing for insolvency administrators and analyzing and improving the early warning tool systems so that firms get early support.      TIMELINE  Short-to-Medium Term      KPIs  Outputs • Publication of national regulations adopting the EU Directive on Restructuring and Insolvency (Di- rective EU 1023/2019) • Specialized track for distressed micro and small business as well as natural person entrepre- neurs to file for expedited insolvency proceedings • Number of insolvent startups, natural person entrepreneurs and micro/small firms supported to exit market by expedited liquidation means. • Number of distressed but viable startups receiving financial and nonfinancial support. Outcome • Increase in serial entrepreneurship • Perceived improvements in regulatory environment 11. The EU Restructuring and Insolvency Directive covers preventive restructuring frameworks, discharge of debt and disqual- ifications, and measures to increase the efficiency of procedures was adopted in 2019. While Member States were supposed to transpose the Directive into national law by mid-July 2021, 23 out of 27 Member States notified the Commission that they encountered particular difficulties in implementing this Directive and opted for the extension of the implementation period by one year. Therefore, by mid-July 2022 national legislation should conform with the Directive. Policy Recommendation 2 62 Reform regulations to strengthen entrepreneurship AND investments CATALYZE INVESTMENTS IN STARTUPS       TARGET BENEFICIARIES  Primary beneficiaries are early-stage investors and secondary beneficiaries are startups and innova- tive firms      RATIONALE  Romanian startups rely on self-financing and are unable to scale effectively. Startups, on average, do not access traditional capital markets for scaling up. Instead, they rely on angel investors, VCs (includ- ing corporate venture capital), and/or crowdfunding platforms.      APPROACH  Introduce or strengthen regulations to incentivize private sector investment activity in growth firms. This includes regulations around stock options, angel investment, corporate and venture capital reg- ulations, and crowdfunding. • Enable Stock Options for Employees: Employee ownership and compensation, via stock op- tions, are used as incentives by high-growth firms in many startup ecosystems. Provision of stock options allows cash-strapped startups to compete for talent against larger (but less innovative) companies, by attracting employees with the potential substantial financial gains that may result from a successful exit. Current Romanian legislation is perceived as unfavorable for the creation and use of ESOPs (employee stock options plans). • Angel and (Corporate) Venture Capital Investor Incentives: Venture capital is a critical resource for many startups. This may take a variety of forms, including ‘Angel’ funding and corporate ven- ture capital (CVC).12 In many ecosystems, public policy instruments are thus used to incentivize venture capital; these often take the form of tax incentives (for example, the United Kingdom’s En- terprise Investment Scheme and Seed Enterprise Investment Scheme) and co-investment funds. Current Romanian incentives are perceived as insufficient, and the legislation surrounding busi- ness angel and business angel syndicate investment is also seen as unfavorable and confusing. It is therefore suggested that a clearer package of incentives for early-stage investors is needed. • Crowdfunding: In Romania, as in other ecosystems, crowdfunding is filling a critical gap in ear- ly-stage financing for startups. In line with the EU Directive on Crowdfunding, we suggest devel- oping and publishing equity crowdfunding guidelines or regulations for Romania.       IMPLEMENTATION BODY  The implementation body for this reform is the Ministry of Finance in collaboration with the Ministry of Investment and EU Projects and the Ministry of Entrepreneurship and Tourism.      RISK  This requires coordination between several entities because they are each responsible for a piece of the entrepreneurship agenda. The reforms should also be informed by the private sector, specifically investors and other ecosystem actors. 12. Traditional venture capital (VC) is no longer the only way for startups to raise money. Many corporations are investing in startups, even startups that operate outside of the corporate’s own vertical. Global accelerator 500 Startups publishes regular insights on the state of corporate-startup engagement, which includes CVC (500 Startups 2019). Policy Recommendation 2 63 Reform regulations to strengthen entrepreneurship AND investments      AMOUNT  €400,000 (This is for the clarification and development of clearer guidelines only; potential incentives, such as tax relief for seed investors, are not included in this figure.)      TIMELINE  Short-to-Medium Term      KPIs  Output • A package of clear guidelines and incentives for investors (angel investing, crowdfunding, (corpo- rate) venture capital) and startup employees, including guidance regarding ESOPs. Outcomes • Increase in financing of young firms (policy recommendation 2 64 strengthen entrepreneurship investments vestment turn basic government-funded into tested approved products man- ufactured domestically. be complemented workshops awareness-raising programs disseminate updated guidance on protections. implementation body  sponsoring body morid, implementing uefiscdi.  risks  needs tied longer-term romania’s innovation system. clarifying pro- tection, if can sold, licensed, transferred, critical, but its di- minished reforms system enacted.  amount  €400,000  timeline  short-to-medium term  kpis  outputs publication clear guidelines arising with- organizations outcomes number spinouts formed university-originated startups holding patents intervention 6: incentivize improve access procurement opportunities primary beneficiaries startups, officers managers, authorities agencies. indirect general (via better services). sector create important market many play role stimulating innova- tion. recent have improved awareness, current processes platforms still difficult small firms. streamlining pub- 65 lic procurement, particular attention innovative expand domes- tic while simultaneously helping bring beneficial innovations sector. assistance world bank, national agency has been undertaking significant (world bank 2020b). we recommend conduct- ing rapid analysis policies, platforms, systems identify how (as opposed msmes) integrated system; also seek extent applicants winners tenders. benefit review countries approaching inclusion young within their systems, order common barriers. example, suggest consider whether excluded through use qualification criteria such international organization standardization (iso) certification demnities; bodies over-specify tenders (for assuming legacy tech- nologies approaches, thus inadvertently excluding solutions); large ten- ders broken smaller pieces fall capacity firms; advertised sufficient time allow these respond appro- priately. measure proportion process- es publicized e-procurement fact 100 percent), well percentage respondents successful msmes). data published annually anap 2021), ideally available real-time basis, application programming interface (api) dataset; increasing openness timeliness assist identifying sec- tors able work more effectively smes, hence prac- tices might adopted bodies. furthermore, experimenting novel models challenge prizes — an ‘outcome-led,’ approach-agnostic model used elsewhere drive encourage new suppliers. providing dedicated training procure- ment personnel processes, differences volved working beneficial; could potentially managed posed one-stop “ecosystem hub” (see recommendations below). agency, under gen- eral secretariat government. strategy re- form 66 increased open start- ups smes amongst tender winners. publish adaptation inclusive simplified procedures (fewer steps, reduced taken) courses methods run officials per strategy). all listed portal (target percent) challenges) ‘outcome-led’ (that is, prescribing approach technology) outcome responding awarded 10: transform procurement. 67 policy 3  establish (or hub”) mission quick support critical sensitive win needed  objective  creation new, agile institution centralize ecosystem initiatives un- one agency. romanian entrepreneurial digital high-growth firms, investors, ecosys- tem enablers, diaspora, faces several challenges, lack understanding they differ reliable about whole. sme legislation mix do account young, growth-oriented business models. poor coordination between policymakers stakeholders. despite promising regional clusters, no co- herent voice champion needs. shows wide variety models, differing purpose, structure, type instruments interventions manage (glennie bound 2016). nevertheless, aridi ka- pil (2019) ingredients success agencies, particularly emerging ecosystems romania’s, adaptable mis- sion, technically capable staff, effective governance management structures, diagnostic-based interventions, robust m&e, sustainable funding, strategic partnerships networks. it receive continuous budget, recruit talent; require exceptions salary caps. some specific examples serve template israeli novation authority (a entity — established chief scientist’s office — that administers fund incubator program, extensive programming); la french tech publicly-funded initiative showcase promote ex- changes them, coherence startups); poland’s na- tional center development (ncbir, approximately €3 billion r&d-intensive firms); poland non-governmental es- tablished advocate policymakers); serbia (administers early stage intensive collaborative industry); coadec privately-funded provide kingdom techleap.nl provides programs, tools, resources startups). ruta, joint venture special district medellín, colombia, further interesting example. had strong “bottom up” stakeholders attracted leaders once showed promise. 3 68 post-transition economies, very often agencies auspices ministry charge innovation, science, technolog- ical (poland, croatia, serbia, bulgaria, kazakhstan) given ministries tend stronger nature driven need companies collaboration see opportunity prevent brain drain even gain among diaspora technical talent. most importantly, however, importance having willingness cross-cutting agen- da entrepreneurship. alliances industry econ- omy only natural highly desirable long (starting closing business, bankruptcy, investors regulations, tax credits, etc.) necessary growth healthy inno- vation ecosystem, scaleups, en- abler organizations. would distinction help negate issues capture incumbents (by different cabinet-level champion), promot- science-driven spinouts, ensure remains focused on. first step draft ‘blueprint’ taking con- sideration roles play. overall objective establishing hub institutional framework address structural challenges scale-ups, facilitators support. struc- ture determined after developed. include: 1 — advocate: identifies promotes specialized seed, startup, scale-up phase investors. 2 — think tank: pilots tests relevance ecosystem. design enablers facilitating adoption crea- tion technologies private addressed pillar iii — business environ- plan). 3 — connector: facilitates systematic communication actors tribute integration articulation positions 4 — capacity builder: develops implements enabler ecosystems. above (‘policy recommendation: enablers’) delivered 5 — funder: supports (grants, loans, ven- capital) acts vehicle relevant non-financial>1 percent of revenue)       ALIGNMENT WITH TOP 12 INTERVENTIONS  Intervention 9: Startup to Scaleup through. 19. Indicative costs based on lowest-cost option available through PlugnPlay, a large startup campus in Silicon Valley that has successfully attracted numerous corporates, startups, and international actors (Plug and Play 2022). 91 POLICY RECOMMENDATION 10  INCENTIVIZE INNOVATION TO FOSTER KNOWLEDGE SPILLOVERS INTO THE PRIVATE SECTOR FOUNDATIONAL TIME GOVERNMENT LONGTERM SENSITIVE SUPPORT NEEDED      OBJECTIVE  Foster knowledge spillovers into startups and the private sector by leveraging universities and research and development institutes.       TARGET BENEFICIARIES  Universities, research and development institutes, and startup founders      RATIONALE  The previous recommendation (“Improve Entrepreneurship Education and strengthen the role of Uni- versities in the ecosystem”) sought to stimulate the supply of innovation from public research. Howev- er, if this supply is not met with a corresponding demand from within the private sector, then it is un- likely to result in more innovations being brought to market. This recommendation therefore aims to encourage the demand side of innovation within the private sector, which is currently relatively weak in comparison with other European countries. This recommendation is not intended to address the wider gaps relating of technology adoption and non-R&D innovation that exist within Romania, which we recognize as important, but which are beyond the scope of this report. Despite large investments in R&D infrastructure projects, public sector investments are not translat- ing into private sector innovation, nor is there substantial demand from private sector firms to build upon the innovations of universities and research institutes. As mentioned in the introduction, and discussed in depth in the interlinked Output 1 report, Romanian firms are significantly less likely to in- novate than the EU average. Only 10 percent of Romanian firms introduced a new or significantly im- proved process or service in the previous twelve months, compared with an average of 60 percent for the EU (Eurostat 2018). Moreover, small firms are especially unlikely to innovate in Romania. Large firms are twice as innovative as smaller firms, whereas across the EU the difference is only 30 per- cent. The gap between Romanian firms and their international competitors is seen in all types of in- novation, although it is especially large in the innovation of new products (see figure 6.4): In addition, Romanian firms have a low production of IP. Romania has around 20 patent applications per billion regional gross domestic product (GDP) in purchasing power standards (PPS); this contrasts with Poland, Croatia, and Bulgaria each filing over 30 patent applications per billion GDP, and inno- vation leaders such as Israel filing over 140 patent applications per billion GDP (European Commis- sion 2020a). One important reason for this is the lack of strong linkages between Romanian firms and universities. Various measures of university-industry collaboration and collaboration between innovative SMEs and other organizations (for example, European Commission 2021b) suggest that Romania lags its coun- terparts in this regard. This means that public sector investments in university research are not trans- lated effectively into private sector patents and related innovations. Policy Recommendation 10 92 Incentivize innovation to foster knowledge spillovers into the private sector FIGURE 6.4  Innovation Performance by Type of Innovation 45 Innovation performance index 40 35 30 25 20 15 10 5 0 EU-27 Croatia Netherlands Romania Czechia Poland Bulgaria Estonia Finland Slovakia Portugal Hungary All four types Product only Product and process Organization and marketing Source: Eurostat 2018. Note: Innovation refers to the implementation of a new or significantly improved product (good or service), process, marketing method, or organizational method in business practices, workplace organization, or external relations. EU-27 = the 27 European Union member states excluding the United Kingdom.      APPROACH  Various approaches to building linkages between research organizations and firms are being trialed in the World Bank’s “Supporting Innovation in Romanian Catching-up Regions” program, although most of the interventions piloted in that program are “supply-side” focused. We suggest that several activi- ties can be taken now to stimulate the demand side of this technology transfer, including: INCENTIVIZING R&D IN EARLY-STAGE FIRMS Existing R&D incentives appear to be inadequate in stimulating the necessary activity in firms and are relatively limited in comparison with many EU countries. Total direct government funding of busi- ness R&D, as a percentage of GDP, is well below the European average (OECD and European Com- mission 2021). (As an example, Poland allocated €3 billion of €3 billion in the last programming pe- riod to business R&D support). One important incentive mechanism is R&D tax credits. Current Romanian R&D relief has remained relatively unchanged in recent years and permits a deduction of 50 percent of costs. However, by com- parison many other countries offer much more generous relief: for example, the United Kingdom R&D tax relief scheme permits a “super deduction” of 230 percent of qualifying R&D costs for SMEs. We therefore recommend an expansion of R&D relief. However, although R&D tax credits are effective at stimulating R&D (Bloom, Griffith, and Van Reenen 2002), such schemes still require that firms to fund the R&D themselves, prior to claiming relief. This incentive is thus more helpful for scaling firms than for driving R&D in young startups and other firms that are not yet revenue positive. As was discussed in the Policy Mix (see Part One of this document, section 2.3.5), there is already a bias in the existing mix of instruments towards mature, profitable firms. Increasing R&D tax credits are unlikely to correct this bias and may exacerbate it. For this reason, we recommend that an expansion of R&D credits be accompanied by direct grant funding for R&D, targeted specifically at startups and small firms — which, as noted in Part One of this Policy Recommendation 10 93 Incentivize innovation to foster knowledge spillovers into the private sector report, are often not the primary target of existing instruments. Such grants should be developed with consideration to the various recommendations in Part One: for example, they should have specific fo- cused objectives, low administrative burdens for applicants, low bureaucratic burdens for adminis- trators, a clear logic for the intervention, and robust instrument-level evaluation. This is likely also to require an expansion of the institutional capacity to distribute such funding. In addition, the grants should be coordination with the startup investment fund (see recommendation above: ‘Create a startup fund’) because, as previously noted, grants and equity investment may displace each other to a degree. Because costs for grants and tax reliefs will depend both on the exact design of the instruments, the definition of qualifying R&D and the uptake by firms, we recommend that new instruments are pilot- ed for a limited period in order to gauge demand, with budget allocated for a comprehensive evalua- tion of the schemes. ENCOURAGING CONSORTIA FOR COLLABORATIVE RESEARCH Many countries encourage collaborative research between firms and research organizations using collaborative research grants, which are restricted to collaborative partnerships or consortia. Such collaborations are known to encourage the exchange of both explicit knowledge and tacit knowledge, with university collaborations being associated with innovative, new-to-market products (Belderbos, Carree, and Lokshin 2004; Chun and Mun 2012). Larger research consortia may also be particular- ly helpful in reducing coordination failure and in stimulating ‘pre-competitive’ research or other re- search of low appropriability (that is, difficult for any one firm to capture for its own competitive advan- tage — meaning that there is a reduced incentive for any one firm to invest in its production, even if it benefits the sector as a whole). We suggest that collaboration formation and consortium formation should be encouraged through the creation of a new collaborative research grant scheme, explicitly requiring the involvement of at least one university partner and at least one private sector firm (ideally an SME). The scheme could initial- ly be targeted towards solving specific societal challenges, such as climate change or sustainability, for which open calls for proposals are invited. Responding consortia or partnerships should be led by a private sector firm, in order to ensure a focus on commercial applications. We suggest that universi- ty partners should still have their costs funded at 100 percent, and that private sector partners may potentially receive a subsidy of their eligible costs, depending on the firm size and the nature of the project. As with the action above, this is likely also to require an expansion of institutional capacity in order to administer the scheme. ESTABLISHING KNOWLEDGE TRANSFER SECONDMENTS We recommend that technology- and knowledge-transfer (both explicit and tacit) is further promot- ed through the public funding of knowledge transfer secondments. Such a scheme would subsidize the secondment of members of academic staff into a firm for a duration, with the specific purpose of encouraging knowledge transfer. It could potentially also operate in reverse — that is, subsidizing the movement of an industry researcher into a university or other research institution. Secondments would be time-limited but tailored to the needs of the firm. For example, secondments under the Knowl- edge Transfer Secondments (KTS) scheme funded by the United Kingdom’s Engineering and Physical Sciences Research Council (EPSRC) may last between 6 weeks and 2 years. Proposed secondments should be appraised by a panel, including senior academics, to determine the quality and feasibility Policy Recommendation 10 94 Incentivize innovation to foster knowledge spillovers into the private sector of the proposed project, and the likely business impact that may result from knowledge transfer. The scheme could potentially be managed at the sub-national (regional) level. Establishment of the scheme will need to address regulatory issues within universities that may inhibit the mobility of researchers. INNOVATION VOUCHERS The recommendation above proposes a scheme to encourage Universities and other research organ- izations to make available their research equipment and infrastructure. To complement this, we pro- pose a scheme targeted at startups and other firms to subsidize the cost of research relating to the use of these facilities. This would take the form of an ‘innovation voucher,’ which would use public funds to pay for a portion of R&D costs in qualified research organizations. We suggest that the val- ue of the voucher should be relatively small, with the application process for vouchers being relative- ly ‘light-touch’ to encourage their adoption; for example, the vouchers could pay for up to €10,000 of facilities use and accompanying academic expertise. We suggest that a proportion of matched fund- ing should be requested from firms using the vouchers, in order to prevent abuse.       IMPLEMENTATION BODY  The Ministry of Research, Innovation, and Digitalization — although some of the recommended actions could be administered regionally. This should also be implemented in coordination with the bodies re- sponsible for the Startup Fund.      RISKS  Key risks include lack of good quality firms because many firms are pursuing opportunity entrepre- neurship. Other risks include lack of coordination between several public authorities because the in- novation agenda is fragmented across several entities, including UEFISCDI.      AMOUNT  The following costs are tentative: Activity Estimated Cost (€) Pilot program for enhanced R&D tax reliefs for startups and scaleups 50 million Direct grant funding of startups and young SMEs (for example, grants of €100,000 x 10 firms per 56 million year x 8 regions x 7-year period) Grant programs for collaborative research consortia (for example, grants up to €500,000 x 20 70 million per year nationally x 7) Pilot of knowledge transfer secondment scheme (aiming for 30 movements in 8 regions, per year) 50 million Pilot program for infrastructure access vouchers / subsidies (for example, 250 firms per region over 20 million period) Total 246 million      TIMELINE  Short-to-Medium Term but also needs to incorporate a longer-term view. Policy Recommendation 10 95 Incentivize innovation to foster knowledge spillovers into the private sector      KPIs  Outputs • Number of startups (& SMEs) applying for R&D tax relief • Number and value of grants distributed • Number of startups (& SMEs) participating in collaborative research consortia • Number of individuals participating in knowledge transfer secondments • Number of vouchers awarded • Number of firms participating in research infrastructure access pilot Outcomes • Increase in total R&D activity by SMEs • Increase in number of patents filed by SMEs • Increase in number of new products or services provided by SMEs       ALIGNMENT WITH TOP 12 INTERVENTIONS  Intervention 6: Incentivize Innovation; Intervention 8: Share R&D Infrastructure. 96 POLICY RECOMMENDATION 11  PROMOTE THE DIGITAL ECONOMY FOUNDATIONAL TIME GOVERNMENT LONGTERM SENSITIVE SUPPORT NEEDED      OBJECTIVE  Create a domestic market for startups by promoting use of digital products, services, and platforms.      RATIONALE  Many high-growth startups are digital. However, Romania ranks 27th of 27 EU Member States in the 2021 edition of the Digital Economy and Society Index (DESI), with low internet use and digital skills inhibiting the domestic market for digital firms. While Romania ranks highly on connectivity (10th in the DESI for this component), it ranks lowly on most indicators measured by this index including being 25th on Integration of digital technology in busi- nesses’ activities, with only 17 percent of SMEs selling online and only 33 percent of SMEs having at least a basic level of digital intensity; and 26th on human capital, with only 31 percent of individuals having basic digital skills (compared to EU average of 56 percent), 10 percent of individuals having above basic digital skills (compared to EU average of 31 percent); and 35 percent of individuals hav- ing at least basic software skills (compared to EU average of 58 percent); and 6 percent of firms offer- ing ICT training. Firms and consumers are thus unable to make the most of the opportunities afford- ed by digitalization and e-commerce, still less the next wave of digital innovation relating to “Web3” protocols and distributed ledger technology.       APPROACH AND ESTIMATED COSTS FOR COMPREHENSIVE PACKAGE OF REFORMS  This recommendation is divided into four specific subcategories. The total costs for each subcatego- ry are listed in the table below. Activity Estimated Cost (€) Category 1: Promoting e-commerce platforms 50 – 100 million Category 2: Increasing Digital Skills 50 – 100 million Category 3: Improving Managerial Skills 50 – 100 million Total 150 – 300 million      RISKS  Activities under this intervention could take a long time to develop because they are linked to other factors. While use of e-commerce platforms accelerated during the pandemic, uptake of digital tech- nologies could remain superficial, and Romania may not fully benefit from productivity transforma- tions needed for firm recovery. Developing skills to consume digital technologies and building trust in digital systems are also longer-term activities. Policy Recommendation 11 97 Promote the digital economy      KPIS  Romania’s performance across various pillars of the Digital Economy is measured annually and pub- lished in the Digital Economy and Society Index (DESI). An overarching outcome is Romania’s im- proved ranking on this index. COMPONENT 1: PROMOTING E-COMMERCE PLATFORMS      OBJECTIVE  Increase Romania’s use of transactional technologies vis-à-vis uptake in use of e-commerce platforms       TARGET BENEFICIARIES  Firms selling online and customers buying online      RATIONALE  Romania’s rates of e-commerce — buying goods and services online — are the lowest in Europe. Be- cause e-commerce is a transactional technology (Hallward-Driemeier et al. 2020), it has the poten- tial to improve the inclusion of smaller firms, including startups, and reach new markets in rural and remote areas. Though use of e-commerce increased somewhat as a result of COVID-19, the address- able domestic market for many firms remains limited.      APPROACH  The government can promote use of e-commerce through education and capacity-building pilot, thus building the domestic market for many startups. There should also be a more coherent voice for e-com- merce within government because at present it is unclear where responsibility resides. It is also rec- ommended that Romania conduct an in-depth analysis on the e-commerce landscape to identify reg- ulatory bottlenecks. We note that, in Qatar, the Ministry of Transport and Communications launched an e-commerce portal to serve as a central resource for information, communication, and support for both merchants and shoppers. Policymakers also need to consider whether Romania’s current physical infrastructure (as opposed to digital) promotes physical connectivity between regions and counties. The quality of physical infra- structure impacts business logistics, including shipping logistics for e-commerce.       IMPLEMENTATION BODY  The implementation body for this activity is the Ministry of Research, Innovation, and Digitalization in coordination with the Ministry of Entrepreneurship and Tourism.      RISKS  Romania may not fully benefit from productivity transformations needed for firm recovery because consumption of digital products and services is linked to digital skills and usage of other digital plat- forms including financial services. As reflected in Romania’s DESI ranking, Romania ranks poorly on percentage of population with basic and above basic skills. Policy Recommendation 11 98 Promote the digital economy      AMOUNT  Estimated costs for a pilot program range from €50 million to €100 million      TIMELINE  Short-to-Medium Term      KPIs  Outputs • Percentage of individuals shopping online • Percentage of firms (including MSMEs) selling online Outcomes • Increase in number of firms selling online • Increase in e-commerce turnover COMPONENT 2: INCREASING DIGITAL SKILLS      OBJECTIVE  Increase basic and above-basic digital skills among the Romanian population.       TARGET BENEFICIARIES  Romanian consumers who lack basic or above basic digital skills; digital startups (indirectly)      RATIONALE  Romania has serious digital skills gaps, societally and in the workforce. Less than one third of peo- ple aged between 16 and 74 have at least basic digital skills (56 percent in the EU as a whole), while some 3 percent have basic software skills (EU average: 58 percent) (European Commission 2021a). Increasing digital literacy will increase the addressable domestic market for digital startups (as well as producing a wide range of other benefits, such as improving productivity and expanding access to digital services). The low uptake of platform technologies (including e-commerce) highlights the real- ity that the diffusion of even basic transactional technologies is not automatic, even with the high ac- cess to broadband internet. Users’ digital literacy and the availability of digital skills matter because they determine the extent to which users turn to platform technologies and the extent to which start- ups are able to find domestic markets (Hallward-Driemeier et al. 2020).      APPROACH  We recognize that there have been numerous studies of the issue of digital skills, both from the World Bank Group and other organizations (for example, Melhem and Jacobsen 2021). We are also aware that the Authority for the Digitalization of Romania is coordinating efforts to improve digital skills, and we underscore the importance of this. Following the recommendations of the World Bank Group report Europe 4.0: Addressing the Digital Dillemma (Hallward-Driemeier et al. 2020), we suggest address- ing digital skills at multiple levels. In primary and secondary education, students should be exposed to basic coding and other digital skills as early as primary school. The Romanian Education board should update tertiary education curricula to meet changing skills requirements (for example, including an understanding of “Web3” technologies — see Lentini and Gimenez (2019) and World Bank (2020a) Policy Recommendation 11 99 Promote the digital economy for more detailed analysis of future skills needs). Universities can ensure that their curricula remain relevant with guidance from the advisory council for educational reforms suggested in Recommenda- tion 6. Given the pace of technological change, lifelong learning opportunities should be supplied for all ages, and should include not only technical skills but also skills for adaptability. One example of how to encourage basic skills development in society is seen in Estonia, which estab- lished the National Estonian Digital Skills and Jobs Coalition to decrease the digital skills gap by focus- ing on programs that can train large portions of the population in digital skills. The coalition includes policymakers, service providers, and information technology (IT) training companies. It has developed self-funded and large-scale training initiatives that aim to lessen the digital skills gap. The purpose of these activities is to lower the number of people in Estonia that do not use computers or the Internet to 5 percent (European Union 2021).. Estonia is also embedding digital skills at within primary and sec- ondary education to ensure that future workforce is equipped with basic and above basic digital skills.       IMPLEMENTATION BODY  The implementation body for this activity is the Ministry of Research, Innovation, and Digitalization in coordination with the Ministry of Education, the Ministry of Labor and Social Solidarity and the Author- ity for the Digitalization of Romania (attached to the Ministry of Research, Innovation and Digitization).      RISKS  This activity will take several years to materialize.      AMOUNT  Estimated costs for a pilot program range from €50 million to €100 million..      TIMELINE  Medium-to-Longer Term      KPIs  Outputs • Percentage of individuals with basic digital skills • Percentage of individuals with above basic digital skills • Percentage of individuals shopping online • Percentage of individuals banking online Outcome • Increase in use of Internet services • Increase use of digital platform Policy Recommendation 11 100 Promote the digital economy COMPONENT 3: IMPROVING MANAGERIAL SKILLS      OBJECTIVE  Upgrading managerial capabilities to promote adoption and use of digital technologies can contribute to the use of transactional technologies20 to contribute to the development of Romania’s digital economy.       TARGET BENEFICIARIES  SMEs      RATIONALE  Managerial practices are important for digitalization: harnessing the benefits of digital technology needs complementary capabilities, and the firms that benefit the most from digital technologies are those that also have better access to key technical, managerial, and organizational skills (OECD 2019a). Low managerial quality, lack of ICT skills and poor matching of workers to jobs curb digital technology adoption and hence the rate of (Andrews, Nicoletti, and Timiliotis 2018). IT is complementary to man- agement practices (and that, because small firms typically have worse management practices, the impact of IT may be less for such irms) (Bloom, Sadun, and Van Reenen 2012). In short, firms that have better management practices are more prone to adopting digital technologies and doing it faster.      APPROACH  The government should pilot a managerial skills training alongside digital education for SMEs to in- crease Romanian firms’ productivity. The government can also introduce incentives for collaboration among firms, such as through vouchers for innovation. In the case strengthening managerial capabil- ities to drive performance leveraging digital technologies, incentives should address the main barri- ers for managers to drive firm-level digitalization, namely preferences and behavioral bias, information constraints, weak external and internal incentives, need of complementary investments, coordination problems, and lack of trust. Therefore, programs should on the one hand encourage the assimilation and demand of digital technologies through focusing on managerial capabilities and on the other sup- port the supply of digital technologies for entrepreneurs / managers. Simultaneously, public interven- tions should generate enabling conditions for firm-level digitalization. Overarching programs typically facilitate tailored assistance, and often provide financial support for entrepreneurs. Hence, overarch- ing programs provide tools for complementary investment, both managerial and financial. Main types of interventions include; a) managerial capability / digital maturity assessment tools (conducted by an expert and/or self-assessed) to access tailored support, designing and deploying business plans, etc., b) provision of financial support, such as vouchers and grants for managerial upskilling, software upgrading, test new digital technologies, technical assistance and business advisory services, and en- abling conditions for digital adoption, c) access to awareness campaigns, workshops, events, and d) provision of public goods such as training courses.       IMPLEMENTATION BODY  The implementation body for this activity is the Ministry of Research, Innovation, and Digitalization. 20. Transactional technologies better match supply and demand to facilitate market transactions by lowering information asym- metries; examples include digital e-commerce platforms and blockchain. Policy Recommendation 11 101 Promote the digital economy      RISKS  The uptake of digital could remain superficial and Romania may not fully benefit from productivity transformations needed for firm recovery. Firms need to continuously invest in managerial skills but may lack resources and awareness of how to accomplish this.      AMOUNT  Estimated costs for a pilot program range from €50 million to €100 million.      TIMELINE  Medium-to-Long Term      KPIs  Outputs • Number of firms using digital platforms • Number of firms investing in innovation • Number of managers upskilled on (digital) productivity tools Outcomes • Increase in digital adoption by firms • Improvements in digital uptake by firms       ALIGNMENT WITH TOP 12 INTERVENTIONS 21  Intervention 11: Appoint Chief Technology Officers in Government22; Intervention 12: Build Confidence in Digital. 21. This recommendation is much broader than the related Interventions identified by the Romanian ecosystem. 22. We concur that this is a critical recommendation but there are many other foundational aspects that need to be prioritized given the foundational nature of this activity. 102 REFERENCES 500 Startups. 2019. “Unlocking Corporate Venture Capital: Finding Eurostat (Statistical Office of the European Communities). 2022. Success in the Startup Ecosystem.” https://go.500.co/unlock- “Migration and Migrant Population Statistics.” Statistics Explained. ing-corporate-venture-capital. March 2022. ANAP (Agenția Națională pentru Achiziții Publice). 2021. “ANAP a pub- Future Work Present. 2020. “Startup and Entrepreneur Visas licat raportul „Indicatorii de monitorizare a eficienței procedurilor around the World.” June 15, 2021. https://web.archive.org/ de achiziție publică finalizate prin contract, în anul 2020”.” Press web/20210615003441/https://futureworkpresent.com/start- Release. August 10, 2021. http://anap.gov.ro/web/anap-a-pub- up-visas. licat-raportul-indicatorii-de-monitorizare-a-eficientei-proceduri- Gavriloaia, George-Cristian. 2020. “The Impact of the Brain Drain Pro- lor-de-achizitie-publica-finalizate-prin-contract-in-anul-2020/. cess on Romania: Possible Solutions in Order to Increase Resil- Andrews, Dan, Giuseppe Nicoletti, and Christina Timiliotis. 2018. “Dig- ience.” CES Working Papers XII (4). ital Technology Diffusion: A Matter of Capabilities, Incentives or Glennie, Alex, and Kirsten Bound. 2016. How Innovation Agencies Both?” Economics Department Working Papers Number 1476. Work: International Lessons to Inspire and Inform National Strate- Paris: Organisation for Economic Co-operation and Development. gies. London: Nesta. Aridi, Anwar, and Natasha Kapil. 2019. Innovation Agencies: Cases Grover Goswami, Arti, Denis Medvedev, and Ellen Olafsen. 2019. High- from Developing Countries. Washington, DC: World Bank Group. Growth Firms: Facts, Fiction, and Policy Options for Emerging Belderbos, René, Martin Carree, and Boris Lokshin. 2004. “Cooperative Economies. Washington, DC: World Bank. R&D and Firm Performance.” Research Policy 33 (10): 1477–92. Hallward-Driemeier, Mary, Gaurav Nayyar, Wolfgang Fengler, Anwar Bloom, Nicholas, Rachel Griffith, and John Van Reenen. 2002. “Do Aridi, and Indermit Gill. 2020. Europe 4.0: Addressing the Digital R&D Tax Credits Work? Evidence from a Panel of Countries 1979– Dilemma. Washington, DC: World Bank. 1997.” Journal of Public Economics 85 (1): 1–31. Harvey Law Group. 2022. “Portugal Start-up Visa Program.” April 30, Bloom, Nicholas, Raffaella Sadun, and John Van Reenen. 2012. 2022. https://harveylawcorporation.com/startup-visa-portugal/. “Americans Do IT Better: US Multinationals and the Productivity Lentini, Valeria, and Gregorio Gimenez. 2019. “Depreciation of Human Miracle.” American Economic Review 102 (1): 167–201. Capital: A Sectoral Analysis in OECD Countries.” International Jour- Bone, Jonathan, Juanita Gonzalez-Uribe, Christopher Haley, and Henry nal of Manpower 40 (7): 1254–72. Lahr. 2019. “The Impact of Business Accelerators and Incuba- Melhem, Samia, and Astrid Herdis Jacobsen. 2021. A Global Study on tors in the UK.” BEIS Research Paper Number 2019/009. London: Digital Capabilities. Washington, DC: World Bank Group. BEIS (Department for Business, Energy & Industrial Strategy). NanoGlobals. 2021. “Europe Startup and Entrepreneur Visa Index.” Cirera, Xavier, and William F. Maloney. 2017. The Innovation Paradox: May 14, 2021. https://web.archive.org/web/20210514230001/ Developing-Country Capabilities and the Unrealized Promise of https://nanoglobals.com/entrepreneur-startup-visas/europe/. Technological Catch-Up. Washington, DC: World Bank. OECD (Organisation for Economic Co-operation and Development). Chun, Hyunbae, and Sung-Bae Mun. 2012. “Determinants of R&D 2019a. “Digitalisation and Productivity: A Story of Complementar- Cooperation in Small and Medium-Sized Enterprises.” Small Busi- ities.” In OECD Economic Outlook, Volume 2019 Issue 1, 55–83. ness Economics 39 (2): 419–36. Paris: OECD Publishing. Cirera, Xavier, Jaime Frias, Justin Hill, and Yanchao Li. 2020. A Prac- OECD (Organisation for Economic Co-operation and Development). titioner’s Guide to Innovation Policy: Instruments to Build Firm 2019b. Talent Abroad: A Review of Romanian Emigrants. Paris: Capabilities and Accelerate Technological Catch-Up in Developing OECD Publishing. Countries. World Bank, Washington, DC. OECD (Organisation for Economic Co-operation and Development), Correa, Paulo. 2014. Public Expenditure Reviews in Science, Technol- and European Commission. 2021. “R&D Tax Incentives: Romania, ogy, and Innovation: A Guidance Note. Washington, DC: World Bank. 2020.” Organisation for Economic Co-operation and Development. European Commission. 2017. Specific Support to Romania: Start-Ups, https://www.oecd.org/sti/rd-tax-stats-romania.pdf. Scale-Ups and Entrepreneurship in Romania. Luxembourg: Euro- Plug and Play. 2022. “Co-Working Space in Silicon Valley.” https:// pean Commission. www.plugandplaytechcenter.com/about/office-space/#paloalto. European Commission. 2020a. “European Innovation Scoreboard The Romanian Parliament. 2022. For the Amendment and Completion 2020”. Luxembourg: Publications Office of the European Union. of the Government Emergency Ordinance no. 194/2002 on the European Commission. 2021a. “Digital Economy and Society Index Regime of Foreigners in Romania. Law 22. Adopted 14 January (DESI) 2021: Romania.” Luxembourg: European Commission. 2022. https://legislatie.just.ro/Public/DetaliiDocument/250498. European Commission. 2021b. European Innovation Scoreboard Smartphone ID. 2022. “What Is a Digital Nomad Visa?” https://www. 2021. Luxembourg: Publications Office of the European Union. smartphone-id.com/en/visa-digital-nomade/. European Commission, North-West Regional Development Associa- World Bank. 2011. Romania Functional Review: Research, Develop- tion, North-East Regional Development Association, and World ment and Innovation Sector. Washington, DC: World Bank. Bank Group. 2021. “Supporting Innovation in Catching-up Regions World Bank. 2018. “High-Growth Firms: Facts, Fiction, and Policy in Romania: Intellectual Property Landscape Analysis Report.” Options for Emerging Economies.” Who We Are/News. November Internal Report. 16, 2018. European Union. 2021. “Estonia—Digital Skills and Jobs Coalition.” World Bank. 2020a. Digital Skills: Frameworks and Programs. Wash- Digital Skills and Jobs Platform. April 24, 2021. https://digi- ington, DC: World Bank. tal-skills-jobs.europa.eu/en/about/national-coalitions/estonia-dig- World Bank. 2020b. Romania Public Procurement Strategy Support. ital-skills-and-jobs-coalition. https://www.worldbank.org/en/news/video/2020/09/23/roma- Eurostat (Statistical Office of the European Communities). 2018. nia-public-procurement-strategy-support. “Community Innovation Survey.” Statistics Explained. March 2022. 103 ANNEX A  TOP 12 INTERVENTIONS IDENTIFIED BY THE ROMANIAN ENTREPRENEURIAL ECOSYSTEM Intervention 1: Reform Startup and Investment Related Regulations Specifically: a. Stock options b. Digitization of company formation c. Venture capital regulations d. Crowdfunding e. Angel investment f. Corporate venture capital Intervention 2: Establish a one-stop agency for Startup Ecosystem Its role could include: a. Advocacy b. Think and Do Tank c. Connector d. Capacity-builder e. Funder f. Data-gatherer Intervention 3: Improve Entrepreneurship Education Specifically: a. Entrepreneurship education b. Infuse academia with entrepreneurs c. Expose lecturers to entrepreneurship Intervention 4: Strengthen Ecosystem Enablers Proposed activities: a. Capacity building for enablers b. University collaboration with European Digital Innovation Hubs (eDIHs) / chambers of commerce / business associations c. Organization of a Romania Ecosystem Summit Intervention 5: Create a Startup Fund This entails: a. Establishing the fund b. Cocreating the investment thesis c. Monitoring fund targets Annex A 104 Intervention 6: Incentivize Innovation Including: a. Enterprise sector collaboration b. Tech transfer capability fund c. Clarification of unclear tech transfer legislation d. Promoting in-house research & development (R&D) activities in firms Intervention 7: Implement Startup Visas Introducing: a. Startup visas b. Incentives for digital nomads Intervention 8: Share R&D Infrastructure Including: a. Developing an open access policy b. Mapping and dissemination c. Pilot program for accessibility Intervention 9: Startup to Scaleup through Exports To improve access to markets, activities include: a. Establishing an overseas ‘Landing Pads’ b. Showcasing Romanian startups c. Creating a diaspora entrepreneurs’ network Intervention 10: Transform Public Procurement Reforms to perceived barriers to public procurement opportunities include: a. Training for public sector officials b. Targeting SMEs / startups c. Challenge prizes Intervention 11: Appoint Chief Technology Officers (CTOs) in Government Proposed activities to help the public sector play a critical role in this agenda: a. Funding CTO roles at the city, regional, and national levels b. Building political legitimacy on all levels c. Introducing sandboxes and testbeds to promote experimentation and innovation Intervention 12: Build Confidence in Digital Promote Internet use and market access through: a. E-commerce education b. “Trust in E-commerce” body c. Clarifying ministerial responsibilities 105 ANNEX B  INSTRUMENTS EXAMINED The program included in the policy mix of 50 instruments were as follows. An asterisk (*) indicates that this pro- gram was also included in the functional analysis. Internal Code Program Instrument EEA Romania — SMEs Growth Programme (EEA and Romania — SMEs Growth Programme (EEA and Norway Grants Norway Grants 2014 – 2021) 2014 – 2021) * FININVEST Programul „Ajutoare de stat pentru finanţarea Schema de ajutor de stat avand ca obiectiv stimularea investitiilor proiectelor pentru investiţii” cu impact major in economie, instituita prin Hotararea de Guvern nr. 807 din 2014 * POC1 POC (Programul Operațional Competitivitate) Investiții în departamentele de CD ale întreprinderilor * POC18 POC (Programul Operațional Competitivitate) Schema de ajutoare de stat si de minimis sub forma de investitii de capital de risc pentru IMM-uri * POC2 POC (Programul Operațional Competitivitate) Clustere de inovare * POC3 POC (Programul Operațional Competitivitate) Întreprinderi inovatoare de tip start-up şi spin-off * POC4 POC (Programul Operațional Competitivitate) Întreprinderi nou-înființate inovatoare * POC7 POC (Programul Operațional Competitivitate) Parteneriate pentru transfer de cunoștințe * POCU1 – 3 POCU (Programul Operațional Capital Uman) Romania Start Up Plus * POCU1 – 3 POCU (Programul Operațional Capital Uman) Romania Start Up Nation IMMINVEST IMMINVEST Programul de susținere a Întreprinderilor Mici și Mijlocii — IMM INVEST ROMÂNIA (program aimed at ensuring the liquidity and working capital of SMEs affected by COVID-19) POC10 POC (Programul Operațional Competitivitate) 1.1.3 — FINALIST-IMM POC11 POC (Programul Operațional Competitivitate) 1.1.3 — COMPLEMENT POC12 POC (Programul Operațional Competitivitate) 1.1.3 — RO-EIT POC13 POC (Programul Operațional Competitivitate) 1.1.3 — RO-ESFRI-ERIC POC14 POC (Programul Operațional Competitivitate) 1.1.3 — CENTRE-SUPORT POC15 POC (Programul Operațional Competitivitate) 1.1.3 — CATEDRE-ERA POC16 POC (Programul Operațional Competitivitate) 1.1.3 — TEAMING POC17 POC (Programul Operațional Competitivitate) 1.2.1 Proiect tehnologic inovativ POC19 POC (Programul Operațional Competitivitate) 2.1.1 Îmbunătățirea infrastructurii în bandă largă și a accesului la internet POC20 POC (Programul Operațional Competitivitate) Secțiunea E-guvernare și interoperabilitate POC21 POC (Programul Operațional Competitivitate) Secțiunea E-guvernare și Open Data POC5 POC (Programul Operațional Competitivitate) Atragerea de personal cu competențe avansate din străinătate POC6 POC (Programul Operațional Competitivitate) Proiecte de investiții pentru instituții publice de CD/ universități POC8 POC (Programul Operațional Competitivitate) Dezvoltare reţele de centre CD — Secțiunea I POC8 POC (Programul Operațional Competitivitate) Proiecte de tip CLOUD și de infrastructuri masive de date Annex B 106 Internal Code Program Instrument POC9 POC (Programul Operațional Competitivitate) 1.1.3 — RO-ECSEL POC9 POC (Programul Operațional Competitivitate) Dezvoltarea RoEduNet POCU1 – 3 POCU (Programul Operațional Capital Uman) Diaspora Start Up POCU4 POCU (Programul Operațional Capital Uman) Bursa student antreprenor POCU5 POCU (Programul Operațional Capital Uman) Innotech Student POR1 POR (Programul Operațional Regional) Operațiunea C — IMM sau IMM în parteneriat cu ITT pentru investiţii de transfer tehnologic POR2 POR (Programul Operațional Regional) Operațiunea B — Sprijinirea parcurilor științifice și tehnologice POR3 POR (Programul Operațional Regional) Operațiunea A — Sprijinirea entităților de inovare și transfer tehnologic (ITT) POR4 POR (Programul Operațional Regional) 2.1.A — Microîntreprinderi POR5 POR (Programul Operațional Regional) 2.1.B — Incubatoare de afaceri POR6 POR (Programul Operațional Regional) 2.2.A — IMM PSIC1 Programul 1.2 Resurse umane pentru CDI M1.2.1.1 Sprijinirea pregătirii unei noi generaţii de cercetători PSIC2 Programul 2.1 Transfer de cunoştinţe M2.1.1.1 Parteneriate pentru transfer de cunoştinţe PSIC3 Programul 2.1 Transfer de cunoştinţe M.2.1.1.2 Sprijinirea clusterelor de inovare PSIC4 Programul 2.2 Stimularea investițiilor CDI din M2.2.1.1 Implementationventure capital instrumentsfor CDI partea sectorului privat PSIC5 Programul 2.2 Stimularea investițiilor M2.2.1.2 Sprijinirea cererii de inovare PSIMM2 Programul 1.2. Înființarea de întreprinderi M1.2.1.2. Programul Național Multianual pentru dezvoltarea spiritului antreprenorial printre femeile din sectorul IMM — Femeia antreprenor PSIMM3 Programul 1.2. Înființarea de întreprinderi M1.2.1.2. Programul Național Multianual pentru dezvoltarea spiritului antreprenorial printre femeile din sectorul IMM — Femeia manager PSIMM5 Programul 1.3 Sprijin IMM M1.3.1.2. Programul Româno-Elveţian pentru IMM-uri (Aria tematica 5 PSIMM6 Programul 1.3 Sprijin IMM M1.3.1.3. Programul Național Multianual pentru sprijinirea Meșteșugurilor și Artizanatului — 2019 PSIMM7 Programul 1.3 Sprijin IMM M1.3.1.3. Programul Național Multianual pentru sprijinirea Meșteșugurilor și Artizanatului — 2020 PSIMM8 Programul 1.3 Sprijin IMM M1.3.1.4. Program pentru dezvoltarea și modernizarea comercializării serviciilor și produselor PSIMM9 Programul 1.3 Sprijin IMM M1.3.2.1. Programul UNCTAD / EMPRETEC pentru România PSMM4 Programul 1.3 Sprijin IMM M1.3.1.1. Programul Național Multianual pentru micro-industrializare 107 ANNEX C  DATA CLASSIFICATION The guide used for classification of general and specific objectives was as follows: General Objectives: These are desired Productivity growth, General-purpose instruments that aim to improve the Economy/society impacts or effects of firm upgrading in productivity and performance of existing businesses. outcomes the intervention. Mark existing business, direct objectives only. technology adoption and diffusion Diversification, new Instruments that have an explicit purpose to support new ventures, new markets businesses, strart-ups, new sectors or new markets. Knowledge creation Instruments that have an direct purpose to improve knowledge and research excellence. Jobs, skills, and human Instruments that have a direct purpose to improve the level capital of human capital, such as training, capacity building, improvement of managerial capabilities, fostering science, technology, engineering, and mathematics (STEM) skills, etc. Societal development Instruments that have a direct purpose to improve social outcomes, inclusion inclusion (for example, innovative solutions to serve disadvantaged groups or instruments targeted towards such groups, start-ups for services with social objectives, etc.) Environment, climate Instruments that have a direct objective to improve the change environment (renewable energy, energy efficiency, etc.) Specific Instrument Scope and objective of Research excellence Instruments that have a principal objective to improve objectives the instrument relation the quality of research, usually at universities or research to the objectives of institutes. innovation support. Mark direct objectives Technology transfer Instruments that have a principal objective to facilitate the only. Multiple and science-industry transfer of knowledge from universities, research centers objectives may be collaboration and other knowledge providers to the private sector. selected. If the objective changed during the Business R&D and Instruments that support and incentivize principal R&D in lifetime of the call, fill R&D-based innovation existing businesses. out according to the latest change and Non-R&D innovation, Instruments that support the adoption of new technologies provide an explanation technology adoption/ or innovation equipment, instruments related to product, in the last field of this diffusion process or business model innovation that do not require a section. formal R&D process. Management practices Instruments that have a principal objective to improve the quality of management practices in enterprises. Access to finance Instruments that have a principal objective to improve access to finance for innovation. It is not sufficient that the call provides financing, it must also have measures to sustain such access in the future through financial intermediaries. Export promotion Instruments that have a principal objective to increase exports (including export financing, quality of exports, etc.) Skills formation Instruments that have a principal objective to train workers and capacity building in areas principally linked to the innovation agenda. Entrepreneurship Instruments that have a principal objective to improve the quality and quantity of entrepreneurship — training in elements of entrepreneurship culture (excluding management practices, new business entry, etc.) Annex C 108 Specific Instrument Improving business Instruments that have a principal objective to improve the objectives (cont.) regulatory business climate as it elates to the innovation agenda, environment/business with specific activities, for example, deregulation, process climate simplification, etc. Market access and Instruments that have a principal objective to enter new integration markets through supplier development programs, cluster formation etc. R&D infrastructure Instruments that have a principal objective to create R&D infrastructure. Environment, climate Instruments that have a principal objective to address change climate change or environmental challenges. Government Instruments that are targeted at government improvement technological with technological solutions. innovation, adoption and diffusion Regional development Instruments that have a principal objective to directly reduce regional disparities and support for specific regions and the integration of markets. Has the instrument This field should be used to indicate whether the objective objective changed of the call was changed over the lifetime of the call, and to over the life of the explain why. program (yes/no), and if so, why? 109 ANNEX D  DETAILED FINDINGS OF THE FUNCTIONAL ANALYSIS CONTENTS D.1 Program Origin   108 D.18 Application and Selection Process   116 D.2 Program Justification   109 D.19 Program Database   117 D.3 Portfolio Relationship   109 D.20 Project Closures   117 D.4 Program Objectives   110 D.21 Budget   118 D.5 Alternative Instrument   110 D.22 Program Management   118 D.6 Logic Model   111 D.23 Roles and Autonomy   119 D.7 Inputs   111 D.24 Staff and Training   119 D.8 Activities   112 D.25 Incentives   120 D.9 Products/Outputs   112 D.26 Process Monitoring   120 D.10 Main Beneficiaries   112 D.27 Monitoring and Evaluation D.11 Selection Criteria   113 (Implementation)   121 D.12 Audiences   113 D.28 Program Coordination    121 D.13 Expected Outcomes and Impacts   114 D.29 Institutional Coordination   122 D.14 Monitoring and Evaluation (Design)   114 D.30 Interaction of Jurisdictions D.15 Learning   115 (Awareness)   122 D.16 Solicitations   115 D.31 Interaction of Jurisdictions D.17 Eligibility Criteria and Application (Seriousness)   123 Information   116 D.1 PROGRAM ORIGIN The program origin indicator refers to the formal pro- cess(es) through which an instrument was created. Figure A.D.1  Distribution of Program Origin The origin of a policy instrument must be embedded Scores in processes covered by the rule of law and consistent 7 with the general high-level goals for which it is intend- ed as an intervention. The identification of the problem 6 being addressed and the means to address it must be 5 grounded in actual evidence through a systematic and Frequency rigorous appraisal of the issue and the options to reach 4 the goals set to address it. 3 The scores for this indicator are close to best prac- 2 tices for this small set of Romanian instruments. (See 1 figure A.D.1) As noted earlier, most of these instruments are funded by the operational programmes (OPs) and 0 1 2 3 4 5 developed as part of the formal operational program Score process. The downside of this is that, on occasion, the external origin of the program appears to have influ- Source: World Bank Group enced other design features. Note: A score of 5 indicates best practice. Best Practices • If the program is a continuation of another program, it must be adapted to the current circumstances, and this improvement should be based on a reasonable diagnosis and evaluation of the previous version of the program. Annex D 110 • Programs should be explicitly linked to strategic objectives relevant to the policy area in question. Programs cannot constitute an end in themselves. D.2 PROGRAM JUSTIFICATION The program justification indicator refers to the pres- ence of specific diagnostics of the particular market Figure A.D.2  Distribution of Program Justification or system failure that an instrument is intended to ad- Scores dress. The justification should be detailed in program 5 documentation and reports and provide an analysis or sensible estimation of the size or intensity of the failure 4 and the impacted population. Frequency 3 Justification scores are midrange or less, except for 2 two instruments. The diagnosis of market or system 1 failure can be improved and the evidence for it, when mentioned, could be more specific. It is one of the in- 0 dicators with both a mid-range average and standard 1 2 3 4 5 deviation, as reflected in the distribution below (figure Score A.D.2). Several instruments can improve on this issue. Source: World Bank Group The rationale for some instruments may not be ground- ed in known systemic failures. Some interviewees reported that the primary reason that an instrument exist- ed was because of the direction of the EC, rather than having themselves identified a particular systemic failure. Best Practices • Diagnostic that covers all the aspects of the market or system failure with evidence and rationale for the need of the specific intervention. D.3 PORTFOLIO RELATIONSHIP The portfolio relationship indicator analyzes how a program operates in relation to all other related and rel- evant programs — at both the national and the European level. The scoring on this indicator includes potential conflicts, complementarities, or overlaps with the rest of the policy mix. A coherent science, technology, and innovation (STI) policy mix is characterized by dynamic Figure A.D.3  Distribution of Portfolio and complementary policy portfolios. For example, a Relationship Scores policy portfolio that supports innovation in companies 4 may have instruments that target companies at differ- ent stages of development, with complementary instru- 3 ments as companies evolve in capabilities, size, market Frequency presence, and so on. 2 Several instruments receive good practice scores. 1 (See figure A.D.3.) The fact that the instruments are proposed within the OP and the consideration of their 0 relevance in the OP portfolio explains the good perfor- 1 2 3 4 5 mance observed. There are some instruments whose Score role in the policy mix may be suboptimal. Source: World Bank Group Best Practices • Instrument design should include a diagnostic analysis of potential conflicts, complementarities or overlaps with the rest of the relevant programs Annex D 111 D.4 PROGRAM OBJECTIVES The program objectives indicator focuses on the ob- jectives and goals that connect the instrument to Figure A.D.4  Distribution of Program Objectives desired system-level changes, such as productivity Scores growth, market competitiveness, knowledge crea- 4 tion, and so on. These objectives should be measura- ble and achievable, with concrete targets. 3 Several instruments score at or below mid-range large- Frequency ly due to lack of measurable objectives at the system 2 level. (See figure A.D.4.) Many instruments specify ob- jectives in terms of project or beneficiary achievements 1 rather than the overall effects of the instrument on the innovation system or economy. Some instruments spec- ify objectives in terms of activities or products of pro- 0 1 2 3 4 5 jects rather than changes in the Romanian innovation Score system. Several instruments also have mixed or un- clear objectives. Source: World Bank Group Best Practices • Objectives should be clear and measurable, with concrete targets. • Objectives should be explicitly tied to the system level changes expected from the intervention. D.5 ALTERNATIVE INSTRUMENT The alternative instrument indicator is closely tied to program justification, in the sense that it evaluates Figure A.D.5  Distribution of Alternative whether different approaches were considered that Instrument Scores can tackle the identified failure that an instrument is 4 intended to address. Cost tradeoffs between alterna- tive instrument designs should be considered. Where 3 approaches based on international examples are be- Frequency ing copied, the designs should be adapted with consid- 2 eration of the local context and capacity of national im- plementing bodies. 1 Many instruments did not consider alternative models 0 of intervention for the stated objectives. The scores 1 2 3 4 5 for the consideration of an alternative instrument show Score considerable variability with an average below mid-range Source: World Bank Group (figure A.D.5). Best Practices • A high-scoring program on this indicator should have included a coherent consideration of different ways for government intervention considering both national and international practices; • A high-scoring program should be able to present high-level justification of the selection for the particular approach selected. Annex D 112 D.6 LOGIC MODEL The logic model indicator measures whether programs have an explicit logic model that connects the inputs, Figure A.D.6  Distribution of Logic Model Scores outputs, and expected outcomes of the program with 7 clear and measurable indicators. The logic model23 should be well articulated, and the expected results 6 should be feasible. Logic models should be updated 5 regularly as programs evolve over time. Frequency 4 Most instruments in Romania lack a logic model. (See 3 figure A.D.6) Romanian instruments consistently score worst on the logic model indicator. The logic of interven- 2 tions is articulated at the OP level, which is required by 1 the European Commission (EC), and is also sometimes present at the level of individual projects but not at the 0 actual intervention level. This ties into the weaknesses 1 2 3 4 5 in instrument objective setting and outcome/impact in- Score dicators that are discussed below. Source: World Bank Group Best Practices • The highest score on this indicator is given to instruments with full explicit logic model with high quality components; • If an explicit logic model does not exist, at least an implicit one does; and • The logic model is used and updated regularly. D.7 INPUTS The inputs indicator measures whether program in- puts are explicitly defined and consistent with a log- Figure A.D.7  Distribution of Inputs Scores ic model. They should cover all resources needed to 5 implement the program, including administrative and 4 operational costs. Costs should be monitored through- Frequency out the implementation phase. 3 2 There is only one instrument in Romania that has an 1 above mid-range score on inputs. (See figure A.D.7.) 0 Cataloging that exists is incomplete and focused only 1 2 3 4 5 on obvious resources. It is of little value to compute ad- Score ministrative costs at the instrument level. Source: World Bank Group Best Practices • A high-scoring instrument on this indicator is one where all inputs (including administrative costs) have been clearly identified and budgeted inside the logic model. • Inputs include all or most of the resources needed to achieve the objectives. 23. Also known as a logical framework (“logframe”) or theory of change (ToC). Annex D 113 D.8 ACTIVITIES The activities indicator measures whether all activities needed to achieve the project objectives have been Figure A.D.8  Distribution of Activities Scores cataloged and are consistent with the logic model. 3 Activities should be consistent with inputs and outputs (that is, all activities have a purpose and help to reach the desired outputs). 2 Frequency There are two issues with activities in Romania. One 1 is that many activities needed to implement instruments are not considered in the design and planned in advance. The second issue is that many instrument activities are 0 1 2 3 4 5 confused with those beneficiaries are expected to car- Score ry out in projects. The scores for activities have larger variability. (See figure A.D.8) Source: World Bank Group Best Practices • Instruments with full and explicit catalog of all necessary activities for achieving the desired objectives; • Activities are consistent with inputs and outputs (that is, all activities have a purpose and help to reach the desired outputs). D.9 PRODUCTS/OUTPUTS The products/outputs indicator measures whether all program outputs and products have been explicitly iden- tified. They should be consistent with all activities and outcomes in order for the desired results to be achieved. All outputs should be operationalized and measurable. The overall scores for products and outputs are a bit Figure A.D.9  Distribution of Products/Outputs higher in Romania because most of the indicators that Scores are actually specified are product indicators. (See fig- 6 ure A.D.9) Their connection with outcomes and impacts are not always well articulated. 5 4 Frequency Best Practices 3 • A high-scoring instrument on this indicator should 2 have explicit and complete identification of prod- ucts related to its logic. 1 • Their relation to outcomes and impact should be 0 explicitly stated. 1 2 3 4 5 • Products/outputs of activities are operationalized Score and measurable. Source: World Bank Group D.10 MAIN BENEFICIARIES The main beneficiaries indicator focuses on whether the beneficiaries targeted in the project design are con- sistent with the overall logic of the instrument. Beyond that, they should also be specified in such a way that can maximize the program efficiency. Criteria for targeting and quantitative measures should be provided in a transparent manner. In Romania, beneficiaries are well defined but poorly targeted. The selected instruments score around the mean or above on this indicator (see figure A.D.10). In all cases, beneficiaries are well defined, but maybe missing either clarity of their justification or optimal targeting. What is generally missing is specific tailoring of the beneficiaries Annex D 114 needed that goes beyond the generic sector or size cri- teria (for example, specific targeting of startups or R&D- Figure A.D.10  Distribution of Main Beneficiaries intensive firms within the much larger pool of SMEs). Scores The best-performing instruments on this indicator have 5 explicitly tailored their beneficiaries in order to achieve 4 maximum results. Frequency 3 2 Best Practices 1 • The selection of beneficiaries should be explicit 0 and coherent with the objectives; 1 2 3 4 5 • There should be a clear strategy for reaching the Score target group effectively. Source: World Bank Group D.11 SELECTION CRITERIA The selection criteria indicator refers to whether selection criteria are consistent with both the objectives and the logic model of an instrument. The design of the selection criteria should be such that it will lead to the maximum potential impact on the targeted population. The criteria should be transparent, simple, and easy to understand. Instruments in Romania include information about selection criteria in their design, but the information Figure A.D.11  Distribution of Selection Criteria needs to be more detailed. The scores on this indica- Scores tor are also mid-range and up. (See figure A.D.11). The 5 instruments included in the design at least some spec- ification of what would have amounted to a good appli- 4 cation submission by potential beneficiaries. The ones that score higher included more detailed criteria to use Frequency 3 later in the assessment of proposals. 2 Best Practices • The selection criteria with the highest scores are 1 those that are clear and coherent. • They should lead to reaching the population with 0 greater impact. 1 2 3 4 5 • The criteria should be transparent, simple, and Score easy to understand. Source: World Bank Group D.12 AUDIENCES The audiences indicator refers to whether all impor- tant program stakeholders can provide input into the Figure A.D.12  Distribution of Audiences Scores instrument design and instrumentation process. There 7 should be formal mechanisms in place for stakeholders 6 to provide input into instrument design and implementa- 5 tion. Instruments should account for non-beneficiary stake- Frequency holders and their role for the success of the instrument. 4 3 Several scores on this indicator are on the higher end, 2 showing that there are clear mechanisms for stake- 1 holder consultations for many instruments in Romania. 0 (See figure A.D.12.) The ones with lower scores can use 1 2 3 4 5 the better-performing instruments to learn how to im- Score prove their own procedures. Source: World Bank Group Annex D 115 Best Practices • The best-scoring instruments on this indicator should account for non-beneficiary audiences and their role in the success of the instrument. D.13 EXPECTED OUTCOMES AND IMPACTS The expected outcomes and impact indicator refers to whether outcomes and impacts at the program level are well defined and connected to desired changes at the system level. They should also be connected to meas- urable results and assessment indicators. Impact indicators should be integrated in the broader policy context on a country or regional level. Criteria should be included for tracking the evolution of outcomes that allow for end- ing program participation if it becomes clear that pro- Figure A.D.13  Distribution of Outcomes/Impacts gram objectives will not be met (as opposed to ad hoc Scores closure at the end of a contract or other extrinsic rea- 7 son not related to results). 6 Many instruments in Romania lack measurable and 5 Frequency coherent impact indicators. The overall score of the 4 selected instruments for expected outcomes and im- 3 pacts is the second lowest for any indicator observed 2 in the functional analysis, with an average of 2.33 (see 1 figure A.D.13). The main problem arises from the lack 0 of measurable and coherent impact indicators. An un- 1 2 3 4 5 derlying issue is that the absence of a logic model leads Score to a lack of connection between outputs and outcomes. Source: World Bank Group Best Practices • The expected outcomes and impact should be explicit, coherent, observable and measurable. • They should be integrated fully with other policies, inside and outside the sector. • They should be used as a performance measure. • Criteria are included for tracking the evolution of outcomes that allow for ending program participation if it becomes clear that program objectives will not be met (as opposed to ad hoc closure at the end of a con- tract or other extrinsic reason not related to results). D.14 MONITORING AND EVALUATION (DESIGN) The monitoring and evaluation (M&E) design indica- tor checks whether there is an integrated M&E sys- Figure A.D.14  Distribution of M&E (Design) tem embedded in the design phase of the instrument. Scores (See figure A.D.14) The indicators chosen should be high 7 quality operationalization of outcomes and impacts. Data collection methods must be realistic, and in cas- 6 es when external data is used, adjustments should be 5 Frequency made to guarantee that the system would meet the spe- 4 cific needs of the instrument. If M&E for the instrument 3 relies on external M&E systems, adjustments or accom- 2 modation of the system must be made to fit the specif- ic needs and aims of the instrument to avoid distorting 1 the instrument logic to accommodate external bureau- 0 cratic requirements that do not favor its functionality. 1 2 3 4 5 Score A centralized system for tracking instruments is Source: World Bank Group in place in Romania, but its connection with M&E Note: M&E = monitoring and evaluation. Annex D 116 processes is hard to gauge. All but one of the instruments had an above mid-range score. These scores were due mostly to the centralized system that is used for capturing all information for the instruments and the pro- jects they support. The exact adaptation of this system to M&E processes for individual instruments was difficult to gauge. The higher score on this indicator seems somewhat contradictory with the lower scores on objectives and outcomes/impacts that it would be building towards. This score may overestimate the quality of the M&E de- sign at the instrument level. Best Practices • An M&E system that is well connected with the program itself and the quality key performance indicator (KPI). • If M&E for the instrument relies on external, administration-wide or organization-wide M&E systems, specific adjustments or accommodation of the system must be made to fit the specific needs and aims of the instru- ment. Practitioners should avoid distorting the program logic to accommodate external bureaucratic require- ments that do not favor its functionality. • M&E indicators are high-quality operationalizations of outcomes and impacts and are not to be confused with indicators of activities or outputs. • Realistic methods of data collection have been considered for the measurement of indicators. D.15 LEARNING The learning indicator refers to formal learning pro- cesses used for systematic improvements of instru- Figure A.D.15  Distribution of Learning Scores ments. Learning processes should be used to improve 6 instrument design and implementation procedures, with systematic and formal documentation of the changes 5 being made. 4 Frequency 3 Learning is happening in Romania, but it is not be- 2 ing documented systematically or formally. (See fig- ure A.D.15) The scores on this indicator are in the mid- 1 range and slightly above. There was evidence of learning 0 and adjustment from experience and a certain level of 1 2 3 4 5 documentation of what was learned. There was no fully Score implemented knowledge management system, though. Source: World Bank Group Best Practices • The best-scoring instrument on this indicator should apply the learning process in the design of the instru- ment with systematic and formal documentation used for long-term improvement. D.16 SOLICITATIONS The solicitations indicator refers to the processes for launching calls for proposals and whether they are rea- Figure A.D.16  Distribution of Solicitations Scores sonable and consistent with the logic model and ob- 4 jectives. Adjustments on subsequent calls for proposals should be well-documented and made with full justifi- 3 cation aligned with the program objectives. Where ap- Frequency propriate, calls should be published regularly and have 2 consistent, predictable calendars. 1 In Romania, the scores for solicitations show high varia- 0 bility with some instruments showing good practices in 1 2 3 4 5 processes for managing solicitations. (See figure A.D.16.) Score Some of the lower-scoring instruments could learn from Source: World Bank Group Annex D 117 the better-performing ones. However, the connection between the ongoing budget execution issues and the solicita- tion processes may not have been clearly captured in this indicator, overestimating the average performance score. Best Practices • The best-scoring instruments are those where adjustments on the calls of proposals are well-documented, made with full justification aligned with the program objectives. • Where appropriate, calls are published regularly and have consistent, predictable calendars. D.17 ELIGIBILITY CRITERIA AND APPLICATION INFORMATION The eligibility criteria and application Information indicator focuses on the eligibility criteria set to reach the target population. They should be clear and transparent and all needed information should be publicly available. Selection information should be collected and analyzed, including lists of applicants, scores awarded to submit- ted proposals, and other pertinent information related to the submission and selection process. This information should be made available to applicants, as much as general privacy regulations allow. The dissemination of infor- mation on eligibility and selection and is consistent with the target population should be appropriate. In Romania, lower scores on eligibility criteria and ap- plication information reflect difficult or burdensome ap- Figure A.D.17  Distribution of Eligibility Criteria plication processes. The score for eligibility criteria and and Application Information application information is one of the lowest in the imple- 3 mentation area at 2.78 but shows great variability (see Frequency figure A.D.17). This means, as we see in the distribution, 2 that several instruments have above mid-range scores. 1 The lower scores reflect a difficult or burdensome applica- 0 tion process that may require that applicants seek exter- 1 2 3 4 5 nal help (such as external consultancies or agents) to sat- Score isfy the application requirements, which imposes a cost. Source: World Bank Group Best Practices • Instruments with transparent selection process, coherent with the target group; • Instruments where the gathered information is used to improve and adapt future selection criteria; • The candidates selected and the amount of money assigned to them are known to the public; and • The administrative burden does not lead to a need for additional resources to submit applications. D.18 APPLICATION AND SELECTION PROCESS The application and selection process indicator refers to the mechanisms for project selection used by the Figure A.D.18  Distribution of Selection Processes implementing body, which should be agile, transpar- Scores ent and responsive. The committees responsible for 5 award decisions should be composed of relevant, inde- pendent experts appointed in a justified and transpar- 4 ent manner. The mechanism for appealing award deci- Frequency sions should be accessible and clear. 3 2 The average score of this indicator is from the mid- range and higher. Several instruments had agile, trans- 1 parent and professional selection processes. (See fig- ure A.D.18.) Several were in the mid-range, having some 0 1 2 3 4 5 weaknesses in the timeliness of the selection process, which was vulnerable to the effects of appeals that Score changed the entire ranking of applications. Source: World Bank Group Annex D 118 Best Practices • The best scoring instruments should have transparent, agile, and responsive selection processes; • Clear appeal and conflict resolution channels should be in place; and • The award decisions should be fully justified and the committees should be set up with the best practices regarding expertise and conflicts of interest. D.19 PROGRAM DATABASE The program database and information indicator refers to the presence of database systems that track par- ticipants, projects, follow-ups, outputs and other data relevant to the program. The system should be used to make adjustments of the solicitations themselves, increase responsiveness to participants’ concerns, and con- tribute to the general improvement of management and design of the program, and should be usable by other programs across the portfolio. In Romania, Instruments mostly rely on the centralized data system, but it is not clear how well the system is tailored to the specific needs of each instrument or whether the information in the central database was used to its full extent. (See figure A.D.19) The program database indicator has a high average score but also high variability reflecting some unevenness in the con- Figure A.D.19  Distribution of Program Database sistency with which information about the progress of Scores projects was either available or used by the instrument 3 managers. They mostly rely on the centralized data sys- Frequency tem, which contains all project information and report- 2 ing. The degree to which it was tailored to the specific 1 needs of each instrument was not clear in all cases. In addition, it was not clear whether the information from 0 the program database was used to its full extent in un- 1 2 3 4 5 derstanding how to make adjustments to the instrument Score or the solicitations. Source: World Bank Group Best Practices • The best scoring instruments should have information and a follow up system in place and the data gathered should be used in order to improve the performance of the program. • The system is used to make adjustments to the calls themselves, to increase responsiveness to participants’ concerns, and to contribute to the general improvement of program management and design. It is also usa- ble by other programs. D.20 PROJECT CLOSURES The project closures indicator focuses on the pres- ence of beneficiary completion or closing reports. The Figure A.D.20  Distribution of Program Closure information obtained from it can be used for learning Scores and improving the impact of the program. 10 Romania scores perfectly on this indicator. This perfect 8 Frequency score (see figure A.D.20) reflects the clear and transpar- 6 ent regulation of the OP on termination of projects that 4 are not meeting requirements of their contract. 2 Best Practices 0 • Closure criteria and reporting should be fully spec- 1 2 3 4 5 ified with high quality reports and post-closure Score results. Source: World Bank Group Annex D 119 D.21 BUDGET The budget and financial resources indicator refers to whether resources are appropriate for the implementa- tion of the program. There should be accountability in the executing of the program and subcontracting entities. In Romania, this indicator has a low average score below the mid-range of 2.67. (See figure A.D.21.) Sev- Figure A.D.21  Distribution of Budget Scores eral instruments have had problems in executing their 6 budgets in a timely fashion and estimating correctly the demand for the particular focus of the intervention. 5 4 Frequency Best Practices 3 • The best scoring instruments have adequate finan- 2 cial resources with proper control and execution mechanisms in place. 1 • Budgets are estimated accurately for the potential 0 demand for the intervention. 1 2 3 4 5 • Program execution and subcontracting entities are Score subject to accountability standards. Source: World Bank Group D.22 PROGRAM MANAGEMENT The program management and organization quality indicator looks at whether the organizational structures in place are appropriate for administering the instruments. The organizational structure should ensure minimi- zation of external and internal pressures in the implementation of the program and should be reviewed for func- tional adequacy given changing requirements with new policies and instruments. In Romania, using a division of labor based on manag- ing authorities (MAs) and implementing bodies may Figure A.D.22  Distribution of Organization impose costs. The management and organization qual- Quality Scores ity indicator has a bimodal distribution, with two pro- 5 grams having perfect scores and the rest mid-range or below. (See figure A.D.22.) The higher scores are due 4 to the involvement of institutions such as the European Economic Area (EEA) and Norway Grants Program that 3 Frequency carry out the implementation of the instrument. The rest are housed in the ministries under regular public ser- 2 vice regimes. As required by the OP, some use a division of labor involving MAs and implementing bodies that 1 are not always aligned with needs of the instrument. Although interviewees generally reported that this divi- 0 sion of labor was usually well-handled, it seems clear 1 2 3 4 5 that this structure imposes some costs and inefficien- Score cies on the overall process. Source: World Bank Group Best Practices • The organizational structure does not have too many levels that make it difficult for information to flow to the implementers • The organizational structure ensures minimization of external and internal pressures in the implementation of the program • The organizational structure is reviewed for functional adequacy given changing requirements with new pol- icies and instruments. It is done using organizational effectiveness indicators. Annex D 120 D.23 ROLES AND AUTONOMY The roles and autonomy indicator evaluated the lev- el of autonomy that implementers had to introduce Figure A.D.23  Distribution of Roles/Autonomy changes and whether clear roles were in place regard- Scores ing program implementation. The implementing body 4 should have the capacity to introduce changes and be capable of resolving conflicts and responding to signifi- 3 cant changes in the political or economic environment. Frequency Romania exhibits mixed performance on this indica- 2 tor. This indicator also shows a bimodal distribution, in part due to the same reason given before (higher scores 1 for the involvement of foreign implementers) but also due, possibly, to different internal arrangements running 0 collections of instruments (see figure A.D.23). Some of- 1 2 3 4 5 fices are run better than others and this is reflected in Score the perception of program managers. Source: World Bank Group Best Practices • The best-scoring instrument should maintain a clear definition of roles to eliminate stress and operate legit- imately; and • Mechanisms for managing change and conflict resolution should be in place. D.24 STAFF AND TRAINING The staff and training indicator evaluates the level of training and experience of the implementing staff. Figure A.D.24  Distribution of Staff/Training This includes assessing whether the number of staff is Scores adequate to implement the program. 3 Romanian instruments scored from the bottom to the top. This indicator has an enormous variability with an average score of 3, exactly in the middle. (See figure 2 Frequency A.D.24.) The heterogeneity in the perceptions of op- portunity for skill enhancement and the adequacy of the staff and skills may be due to local conditions with- 1 in the ministries. Some of these issues are difficult to gauge with videoconferencing interviews that do not al- low for direct observation of the work conditions. Some 0 instruments did not seem to have enough skilled staff 1 2 3 4 5 nor many opportunities for improving their own staff. Score Others thought they were sufficient. Source: World Bank Group Best Practices • The best-performing instruments on this indicator have staff with exceptional experience and high-quality educational background; and • There are formal training opportunities and incentives for advancement. Annex D 121 D.25 INCENTIVES The incentives indicator refers to the presence of clear and explicit criteria for assessing staff performance. Figure A.D.25  Distribution of Incentives Scores Awards and punishments should be linked to clear es- 4 tablished criteria and related to the individual’s duties and the performance of the instrument. 3 Frequency The average score on the incentives for performance and their relation to the instrument outcomes is low 2 and reflects a separation of the staff performance evaluation from the performance of the instruments 1 they manage. (See figure A.D.25) Staff performance fol- lows standard hierarchical procedures with evaluations 0 down the chain of command by the direct supervisor. 1 2 3 4 5 However, several staff commented that performance Score management was “all stick and no carrot.” Source: World Bank Group Best Practices • Instruments are managed by staff that are being evaluated based on well-specified criteria. The management has clear mechanisms for rewarding good and punishing bad performance, tied to the successful achieve- ment of program goals. D.26 PROCESS MONITORING The process monitoring indicator checks for the pres- ence of an audit and monitoring system with clear Figure A.D.26  Distribution of Process Monitoring indicators for monitoring program implementation. Scores This system should be applied periodically, and reports 4 should be submitted and presented to higher authori- ties. Monitoring should aim for effective oversight with- 3 out becoming oppressive. Frequency 2 Except for the two instruments with higher scores, most are in the mid-range or below. (See figure A.D.26.) The 1 latter group of instruments are subject to the standard- ized audit procedure on a national level. Simultaneous 0 audits seem to be problematic. That is to say, in some 1 2 3 4 5 cases the lower scores were due to the perceived bu- Score reaucracy of too much monitoring, rather than too little. Source: World Bank Group Best Practices • Instrument subject to robust administrative process monitoring with quality measurement of indicators. • The process should appear periodically and inform timely decisions from management. Annex D 122 D.27 MONITORING AND EVALUATION (IMPLEMENTATION) M&E in implementation refers to the presence of a formal M&E system that operates continuously and is used to modify programs and generate impact evaluations. The information should be collected at all indica- tors levels and the latter should be improved with time. There must be an impact assessment, as well as mech- anisms for learning and adapting the program. Programs should be revised based on implementation lessons. The implementation of M&E has high variability and an average score above the mid-range. (See figure A.D.27.) This is partly due to the instrument managed by a foreign agency that has a perfect score. The rest of the scores also seem high when compared to practices in peer coun- tries using similar M&E infrastructure. More emphasis was put on the existence of the centralized digital infra- Figure A.D.27  Distribution of M&E structure than on specific M&E practices for continuous (Implementation) Scores improvement. This score is probably overvalued. 4 Best Practices 3 Frequency • The best performing instrument should have a for- mal high-quality M&E program that operates con- 2 tinuously and is used to modify programs and gen- 1 erate impact evaluations. • Information is collected for indicators at all levels 0 (activities, products, etc.). Indicators are adapted 1 2 3 4 5 and improved with time. Score • Programs have been revised based on implemen- Source: World Bank Group tation lessons. Note: M&E = monitoring and evaluation. D.28 PROGRAM COORDINATION The program coordination indicator refers to wheth- er a program acknowledges the existing portfolio of Figure A.D.28   Distribution of Program STI programs and has established good coordination Coordination Scores and communication with other programs. In cases 5 when overlaps occur, they should be resolved as pro- grams are combined or coordinated. Explicit comple- 4 mentarity criteria should exist for the overall effective- Frequency 3 ness of related programs. 2 The Romanian agencies manage collections of instru- 1 ments within the OP and have built-in mechanisms of coordination among them. As a result, the average 0 score is relatively high, above the mid-range. (See figure 1 2 3 4 5 A.D.28.) There was not much clarity about the strategic Score significance of this coordination mechanism. Source: World Bank Group Best Practices • The best-performing instruments are those that acknowledge the existing programs and have established good coordination and communication. • In cases when overlaps occur, they are resolved as the programs are combined or coordinated. • Explicit complementarity criteria exist for the overall effectiveness of related programs. Annex D 123 D.29 INSTITUTIONAL COORDINATION The institutional coordination indicator refers to coordination and participative mechanisms with other pub- lic and private institutions. Evidence of joint work with other institutions should be in place. In Romania, the coordination of institutions is a prod- uct of the similar mechanisms among MAs and im- Figure A.D.29   Distribution of Institutions plementing bodies that are required by the OP. (See Coordination Scores figure A.D.29.) The question of how this coordination 4 results in improvement of design for future cycles, tak- ing into consideration synergies and complementari- ties, remains unclear. 3 Frequency Best Practices 2 • The best performing instruments are those that have established formal coordination mechanisms 1 with evidence of collaborative work and design. • Joined and coherent strategic management should be present. 0 1 2 3 4 5 • There needs to be evidence showing that work Score and joint design processes take place with other institutions. Source: World Bank Group D.30 INTERACTION OF JURISDICTIONS (AWARENESS) The internal interaction of jurisdiction rules and regulations indicator refers to the implementing staff’s knowl- edge of laws and regulatory constraints. Programs should be adapted based on jurisdictional limitations, and program staff should be capable of taking action to lev- erage positive or mitigate negative factors. Figure A.D.30   Distribution of Interaction of For the most part, program managers are aware of Jurisdictions (Awareness) Scores jurisdiction rules that affect the potential for perfor- 5 mance of the instruments they manage. (See figure A.D.30.) State-aid regulations are especially problem- 4 atic and difficult to work around. Frequency 3 Best Practices • The best scoring instruments are the ones where 2 the implementing body is fully aware of and 1 acknowledges the jurisdiction interactions. • The best scoring instruments are the ones where 0 the implementing body has taken action to adapt 1 2 3 4 5 the program based on the jurisdiction rules and Score regulations for optimal performance. Source: World Bank Group Annex D 124 D.31 INTERACTION OF JURISDICTIONS (SERIOUSNESS) The interaction of jurisdiction rules and regulations (seriousness of external constraint) indicator refers to the degree to which the general legislative and regulatory environment inhibits program implementation and effectiveness and the degree to which any regulatory obstacles are modifiable. For some instruments in Romania, the state-aid reg- ulations were a difficult obstacle to overcome, as re- Figure A.D.31   Distribution of Interaction of flected in the instruments with lower scores. (See Jurisdictions (Seriousness) Scores figure A.D.31.) Except for those run by a foreign or ex- 4 ternal agency, the ones with higher scores may reflect a misperception of how the obstacles work, simply ac- 3 Frequency cepting conditions uncritically. Those scores seem too high, given experiences elsewhere in the region. 2 1 Best Practices • Degree to which general legislation or regulation 0 can leverage the program impact. 1 2 3 4 5 • Degree, likelihood, and timeliness to which these Score obstacles are modifiable. Source: World Bank Group 125 ANNEX E  LIST OF COUNTRIES WITH A STARTUP VISAS AND/OR DIGITAL NOMAD VISAS   24 Country Program Program Type Australia Business Innovation and Investment (Provisional) Subclass 188 (Entrepreneur Stream) Entrepreneur Austria Red-White-Red Card for Startup Founders Startup Canada Start-up Visa Program Startup Cayman Islands Cayman Enterprise City (CEC) Entrepreneur Chile Start-Up Chile Startup, investor Cyprus Cyprus Startup Visa Scheme Startup Denmark Start-up Denmark Startup Estonia Estonia Digital Nomad Visa Digital Nomad Estonia StartUp Visa Estonia Startup Finland Start-up Entrepreneur Residence Permit Startup France French Tech Visa Startup Georgia Remotely from Georgia Digital Nomad Germany Self-Employed Residence Visa Entrepreneur Ireland Start-up Entrepreneur Programme (STEP) Entrepreneur Israel Innovation Visas Program for Foreign Entrepreneurs Entrepreneur Italy Italian Startup Visa Startup Japan Startup Visa Startup Latvia Latvia Startup Visa Startup Lithuania Startup Visa Lithuania Startup Malaysia  Malaysia Tech Entrepreneur Programme (MTEP) Entrepreneur Netherlands Dutch Startup Visa Startup New Zealand Entrepreneur Work Visa Entrepreneur New Zealand Global Impact Visa Entrepreneur Portugal StartUP Visa Startup, investor Rwanda  W2 — Entrepreneur in Information Technology and related activities Entrepreneur San Marino San Marino Startup Visa Startup Singapore  EntrePass  Entrepreneur Spain Entrepreneur Visa Program Entrepreneur Sweden Self-Employed Residence Permit Entrepreneur Thailand  SMART S (Startup) Visa Startup 24. Based on data from Future Work Present (2020) and NanoGlobals (2021). Annex E 126 Country Program Program Type United Arab Emirates Ras Al Khaima Economic Zone Entrepreneur United Arab Emirates Dubai Silicon Oasis Authority Entrepreneur United Arab Emirates Fujairah Creative City Entrepreneur United Arab Emirates DMCC Entrepreneur United Kingdom Innovator Visa Startup United Kingdom UK Start-up Visa Startup FIGURE A.E.1  International Good Practice Guidelines for Developing an Incentive Scheme Such as a Startup Visa Program Pre-Approval Investment timeline Graduating firms A Well-defined policy B Targeted eligibility C Cost-effective incentives D Rigorous monitoring E Clear exit policy objectives instrument and evaluation What are the specific and Is the selection criteria Are incentive instruments Are incentivised firms Are there effective measurable policy goals for incentives approval cost-effective in subject to frequent safeguards that prevent to be pursued through objective and clearly improving profitability review and rigorous abuses through the investment policies and bed to national of firms? monitoring? shifting of incentives? incentives? aspirations? F Administration and governance

Previous article
Next article

Most Popular