Data: Needs and Scoring

WHAT DATA DO YOU NEED?

Often times, data and documentation can show that a management system exists - and is functioning - and yet fail to show whether it is functioning effectively. Similarly, isolated information on fishery performance can be misleading (e.g. several years of declining landings do not necessarily point to a failing management system). However, with the necessary context and a comprehensive assessment it is possible to identify where there is a sound management system in place leading to good performance across a range of fisheries, compared to an apparently well-performing fishery is a one-off that lacks stability and durability. Therefore, it is important to look at data representing both the system and its outcomes. Selected indicators must collectively provide a comprehensive, but manageable, representation of a system's performance.

The FGT requires users to collect and examine documents from across the fisheries governance system. These can be laws, regulations and policies, scientific reports, organizational charts and roles, plans, procedures, and steps to ensure transparency, participation, and evaluation of outcomes. Because the framework can be applied to relevant large and local scale governance systems - international, federal, state-provincial, local-community - it requires users to apply information to a suite of metrics applicable at the desired scale. This may require documentation from one or several levels of governance: country- wide, regional or local. Selected fisheries that operate within the jurisdiction of the local governance body serve as proxies of the functioning of the overall management system. We rely on this approach to provide a representative of fisheries management across the system, without the burden of assessing all fisheries, but recognize that every fishery is unique and its performance is a result of many intervening factors and fishery characteristics.

Wherever possible, evidence that enabling factors are present (across Components 1: Policy and 2: Capacity) will draw from primary documentation, such as laws, regulations, decrees, and policy statements. Evidence may also rely on secondary and summary reports, followed by qualitative and anecdotal evidence from country and fishery experts. Component 3 requires documentary evidence of management system attributes and outcomes such as harvest strategies and stock assessments at a fishery level. Component 3: Performance will represent the fishery strategies, regulations and achieved performance across the triple bottom line for individual fisheries. The critical need across the FGT is the provision of referenced, defendable evidence to justify the evaluation and provide a standardization across users.

Policy and Capacity (enabling factors) evaluate the governance system, at the scale of interest. This may be country or regional and levels. In scoring Policy and Capacity, the indicators require assessment that criteria exist and are implemented across the scales. For example, we see many instances where a national policy devolves management to regional authorities, yet those regions do not receive the support necessary to implement the policy. Performance (fisheries outcomes) indicators are specific to the fishery in assessment but may be measured at various scales including at a fishery, stock, ecosystem, country-wide or community level.

Performance outcomes are typically measured at various scales including at a fishery, stock, ecosystem, country-wide or community level, where relevant (Table 1). This is done to recognize that there are optimal scales that will be most meaningful and that there may be challenges with data availability across those scales. This is important in understanding the information available for assessment, as aggregate data (representing only revenue at a national scale, for example) may mask the local or regional impact that a fishery's revenue may have.

The evaluation process is conducted largely as a desk-top study but requires knowledge of the system in question and the ability to confirm information with relevant offices and contacts. For many countries, detailed information may not be readily available online and will require additional steps to attain. Evaluations may rely (in part) on available credible sources that already compile information on fishery performance (such as MSC certifications, Seafood Watch reports, FishSource profiles, and related), but application of compiled reports should be reviewed prior to their use in this context, particularly with respect to the scale of assessment.

HOW IS PERFORMANCE SCORED?

The FGT framework is designed at a yard stick against which progress in a country's policy development, policy implementation, and performance outcomes can be measured and tracked over time. Country assessments are based on information that demonstrates measurement against numerous indicators. It is intended to identify progress rather than prescribing a "standard" that must be met. It requires evidence and serves as a diagnosis of whether strategies and theories of change are working, whether elements of the governance framework support each other, and where gaps or barriers might exist.

Within each indicator, the assessor considers whether the governance system of fishery 'meets' the measure as evaluated across a range of performance that demonstrates progression (example from Component 1 provided in Table ). Each of the more than 200 measures in the FGT framework is evaluated and assigned a nominal score: Basic, Adequate, Good, or Better. Measures of Basic and Adequate are essential for establishing sound and durable fisheries management, while Good and Better measures will promote more sustainable management. Achieving a measure in "Better" does not mean the system is "perfect". A measure scored as Better in the absence of related elements at the Basic or Adequate level may in fact undermine the potential benefit of measures at the Good or Better level.

The idea of the nominal scores is that as fisheries management becomes more sophisticated there is a diminishing return with regards to outcomes (see Scoring the Quality of Supporting Evidence below). Enabling factors at the Basic and Adequate levels are most important for underpinning fisheries management. The FGT does not prescribe a standard or require that every system should meet all measures. Its function is to assess the existing governance system performance, then measure progress and change that can be accomplished within the context of where stakeholders aspire to be.

Does the country 'meet' the measure? What gets a Yes?
Each of the measures is scored independently with a response of "Yes," "No," "In Part," "Not Evaluated," or "Not Applicable." (Note: "Not Applicable" is only relevant to individual fishery assessment under Component 3). If the evidence supports a "Yes" answer that the measure is met, it gets the full score possible for that measure. If the evidence supports an "in-part" answer, it gets half the score. Recognizing that scoring may be done by individuals or groups with varied expertise, and that individuals bring their own level of knowledge and perspective to the task, it is important to define as precisely and clearly as possible a standard of evidence required to get to "Yes: The country meets the measure."

To provide a level of quality assurance and control that all scores are based on similar (if not precisely equivalent) evidence, we provide specific guidance in Appendix C that includes guidance on what is required for each score to be met, and examples of the kinds of documentation that may be used across each of the three Components.

DETERMINING QUALITY OF SUPPORTING EVIDENCE

Accompanying the indicator data scores is an index of information quality - ranging from best data to no data - to provide an estimate of information uncertainty. Sources of data and information include peer reviewed literature, gray literature, expert opinion, anecdotal experience, and traditional ecological knowledge that can be verified. Data quality indices have been applied elsewhere in evaluating the quality of information in analyses¹.

Ranging from best data to no data, each measure is assigned a data quality index to provide an estimate of information uncertainty. Scoring gaps (those measures that could not be evaluated) should also be reflected in the data quality score. Scoring the quality of the data provides an additional lens through which the results should be considered - such as where a perceived well-performing system is determined based on dated information.

__________________________________
¹For example, see the index employed in Productivity and Susceptibility Analysis methods of Patrick et al. (2010).