KPIs for Australian Public Mental Health Services: PI 01 – Change in consumer's clinical outcomes, 2015–

Identifying and definitional attributes

Metadata item type:Indicator
Indicator type:Indicator
Short name:MHS PI 01: Change in consumer's clinical outcomes, 2015–
METeOR identifier:584024
Registration status:Health, Standard 19/11/2015
Description:

The proportion of episodes of mental health care where:

  • significant improvement;
  • significant deterioration;
  • no significant change;

was identified between baseline and follow-up of completed outcome measures.

NOTE: This indicator is related to MHS PI 01J: Change in consumer's clinical outcomes (Jurisdictional level version). There are technical differences in the consumer groups listed in the methodology section between the Service level version and the Jurisdictional level version of this indicator. Caution should be taken to ensure the correct methodology is followed.

Rationale:

Mental health services aim to reduce symptoms and improve functioning. Their effectiveness can be compared using routinely collected measures. This will assist in service benchmarking and quality improvement.

The implementation of routine mental health outcome measurement in Australia provides the opportunity to monitor the effectiveness of mental health services across services and jurisdictions.

Indicator set:Key Performance Indicators for Australian Public Mental Health Services (Service level version) (2015–) Health, Superseded 16/02/2021

Collection and usage attributes

Computation description:

Coverage/Scope:

All public mental health service organisations.

The following episodes of care are excluded:

Methodology:

  • Only the Health of the Nation Outcome Scales (HoNOS) family of measures (including HoNOS, HoNOS65+ and HoNOSCA) is considered in the calculation of this indicator.
  • Outcomes are calculated for the following consumer groups:
    • Group A: Consumers discharged from hospital. All people who were discharged from an acute psychiatric inpatient unit within the reference period. Scores should be calculated as the difference between the total score recorded at admission (the ‘baseline’) and discharge (the ‘follow-up’).
    • Group B: Consumers discharged from ambulatory care. All people who were discharged from an ambulatory care episode within the reference period. Scores should be calculated as the difference between the total score recorded at admission to the episode (the ‘baseline’), and discharge from the episode (the ‘follow-up’). Ambulatory episodes that are completed because the consumer was admitted to hospital or residential mental health care must be excluded from the analysis that is, where the National Outcomes Casemix Collection (NOCC) ‘reason for collection’ equals change of setting.
    • Group C: Consumers in ongoing ambulatory care. All people who have an ‘open’ ambulatory episode of care at the end of reference period. Scores should be calculated as the difference between the total score recorded on the first occasion rated within the reference period which will be either admission or review, (the ‘baseline’) and the last occasion rated which will be a review (the ‘follow-up’) in the same reference period.
    • Group D: Consumers discharged from residential mental health care. All people who were discharged from a residential mental health service unit within the reference period, excluding statistical separations. Scores should be calculated as the difference between the total score recorded at admission (the ‘baseline’) and discharge (the ‘follow-up’).
  • Group change analyses can only be determined for episodes of care where both baseline and follow-up ratings are present. This excludes specific episodes defined by the NOCC data collection protocol as not requiring follow-up as well as episodes where either the baseline or follow-up measure is not available.
  • The total score is determined for each individual baseline and follow-up score. This is the sum total of the 12 HoNOS/65+ scales or the first 13 items of the 15 HoNOS Children and Adolescents (HoNOSCA). Where one or more of the HoNOS/65+ or HoNOSCA items have not been completed correctly, the collection occasion should only be regarded as valid and complete if:
    • For the HoNOS and HoNOS65+: A minimum of 10 of the 12 items have a valid severity rating (i.e. a rating of either 0, 1, 2, 3 or 4);
    • For the HoNOSCA a minimum of 11 of the first 13 items have a valid severity rating;
    • There are no corrections made for missing items.
  • Scores are classified as either ‘significant improvement’, ‘significant deterioration’ or ‘no significant change’, based on the effect size statistic.
  • The reference period for this indicator is typically a single financial year, and the impact of modifying the reference period is unknown.

Definition notes:

  • For the purposes of this KPI, a medium effect size of 0.5 is used to assign outcome scores to the three outcome categories. A medium effect size is equivalent to an individual change score of at least one half (0.5) of a standard deviation.
  • Individual episodes are classified as either: ‘significant improvement’ if the effect size index is greater than or equal to positive 0.5; ‘significant deterioration’ if the effect size index is less than or equal to negative 0.5; or ‘no change’ if the index is greater than -0.5 and less than 0.5.
  • Analyses of the 2010–11 national pool of NOCC data identified that for:
    • Inpatient and ambulatory settings a four point threshold indicates a medium effect size. In practice this means an individual change score of a least four points equates to a medium effect size and significant change. This threshold is the same for all three HoNOS measures.
    • Residential settings a five point threshold indicates a medium effect size. In practice this means an individual change score of a least five points equates to a medium effect size and significant change. There was only sufficient data to construct for the HoNOS measure.
Computation:

(Numerator ÷ Denominator) x 100

Calculated separately for each group.

Numerator:

Number of completed or ongoing episodes of mental health care with completed outcome measures, partitioned by setting, where significant improvement/significant deterioration/no significant change was identified between baseline and follow-up within the reference period.

Denominator:

Number of completed or ongoing episodes of mental health care with completed outcome measures, partitioned by mental health setting within the reference period.

Disaggregation:

Service variables: Target population (adult only)

Consumer attributes: Diagnosis, age, SEIFA, remoteness, Indigenous status

Representational attributes

Representation class:Percentage
Data type:Real
Unit of measure:Service event

Indicator conceptual framework

Framework and dimensions:Effective

Accountability attributes

Benchmark:

Levels at which indicator can be useful for benchmarking:

  • service unit
  • mental health service organisation
  • regional group of services
  • state/territory.
Further data development / collection required:

The NOCC maintained by the Department of Health compiles all state and territory consumer outcomes data on an annual data submission basis. However, NOCC does not allow linkage of episodes of care across financial years.

This limitation does not exist for states and territories own data sets.

A proxy solution using a sub set of episodes, which have commenced within the same financial year, is used for producing this indicator from NOCC. Compared to data constructed from state and territory datasets and using complete episodes, this limitation is likely to result in calculation of the indicator based on a significantly smaller pool of episodes. It may also have a systematic effect on apparent results on this indicator, but the direction and size of this effect has not been systematically tested.

Longer term, a process is needed that allows data reported by states and territories to the NOCC for consumers who begin an episode in a given year to be tracked when the episode continues into subsequent years. Work is underway to build in an episode identifier into NOCC. Additionally, consistent, cross-year use of service identifiers and unique identifiers for consumers is necessary to enable full capacity to construct this indicator.

Other issues caveats:
  • This indicator measures one type of outcome for mental health consumers. Where possible, NOCC-based measures should be complemented by other measures of consumer outcomes (e.g. social outcomes such as employment or social participation) that capture different perspectives on consumer outcomes.
  • Greater variability in HoNOS scores can be expected, if effect sizes were calculated using ‘locally’ derived standard deviations. Statistically, fewer observations generally give rise to greater variability.
  • This indicator was designed as a measure of aggregate group change.

Source and reference attributes

Submitting organisation:

Australian Institute of Health and Welfare on behalf of the National Mental Health Performance Subcommittee

Reference documents:

Key Performance Indicators for Australian Public Mental Health Services, Third edition (2014)

Relational attributes

Related metadata references:

Has been superseded by KPIs for Australian Public Mental Health Services: PI 01 – Change in mental health consumer's clinical outcomes, 2018 (Service level) Health, Superseded 16/02/2021