was identified between baseline and follow-up of completed outcome measures.
NOTE: This specification has been adapted from the indicator MHS PI 01: Change in consumer's clinical outcomes, 2015– (Service level version) using terminology consistent with the National Health Data Dictionary. There are technical differences in the consumer groups listed in the methodology section between the Service level version and the Jurisdictional level version of this indicator. Caution should be taken to ensure the correct methodology is followed.
Rationale:
State/territory specialised mental health services aim to reduce symptoms and improve functioning. Their effectiveness can be compared using routinely collected measures. This will assist in service benchmarking and quality improvement.
The implementation of routine mental health outcome measurement in Australia provides the opportunity to monitor the effectiveness of mental health services across jurisdictions.
Ambulatory episodes of mental health care that end because of admission to hospital or residential mental health care service (‘reason for collection’ equals change of setting).
Methodology:
Reference period for 2015 performance reporting: 2013–14.
Only the Health of the Nation Outcome Scales (HoNOS) family of measures (including HoNOS, HoNOS65+ and HoNOSCA) is considered in the calculation of this indicator.
Outcomes are calculated for the following consumer groups:
Group A: Consumers discharged from hospital. All people who were discharged from an acute admitted patient mental health care service within the reference period. Scores should be calculated as the difference between the total score recorded at admission (the ‘baseline’) and discharge (the ‘follow-up’).
Group B: Consumers discharged from ambulatory care. All people who were discharged from an ambulatory care episode, provided by an ambulatory mental health care service, within the reference period. Scores should be calculated as the difference between the total score recorded at admission to the episode (the ‘baseline’), and discharge from the episode (the ‘follow-up’). Ambulatory episodes that are completed because the consumer was admitted to hospital or residential mental health care are excluded from the analysis that is, where the National Outcomes Casemix Collection (NOCC) ‘reason for collection’ equals change of setting.
Group C: Consumers in ongoing ambulatory care. All people who have an ‘open’ ambulatory episode of care at the end of reference period. Scores should be calculated as the difference between the total score recorded on the first occasion rated within the reference period which will be either admission or review, (the ‘baseline’) and the last occasion rated which will be a review (the ‘follow-up’) in the same reference period.
Group change analyses can only be determined for episodes of care where both baseline and follow-up ratings are present. This excludes specific episodes defined by the NOCC data collection protocol as not requiring follow-up as well as episodes where either the baseline or follow-up measure is not available.
The total score is determined for each individual baseline and follow-up score. This is the sum total of the 12 HoNOS/65+ scales or the first 13 items of the 15 HoNOS Children and Adolescents (HoNOSCA). Where one or more of the HoNOS/65+ or HoNOSCA items have not been completed correctly, the collection occasion should only be regarded as valid and complete if:
For the HoNOS and HoNOS65+: A minimum of 10 of the 12 items have a valid severity rating (i.e. a rating of either 0, 1, 2, 3 or 4);
For the HoNOSCA a minimum of 11 of the first 13 items have a valid severity rating;
There are no corrections made for missing items.
Scores are classified as either ‘significant improvement’, ‘significant deterioration’ or ‘no significant change’, based on the effect size statistic.
The reference period for this indicator is typically a single financial year, and the impact of modifying the reference period is unknown.
Definition notes:
For the purposes of this KPI, a medium effect size of 0.5 is used to assign outcome scores to the three outcome categories. A medium effect size is equivalent to an individual change score of at least one half (0.5) of a standard deviation.
Individual episodes are classified as either: ‘significant improvement’ if the effect size index is greater than or equal to positive 0.5; ‘significant deterioration’ if the effect size index is less than or equal to negative 0.5; or ‘no change’ if the index is greater than -0.5 and less than 0.5.
Analyses of the 2010–11 national pool of NOCC data identified that for:
Inpatient and ambulatory settings a four point threshold indicates a medium effect size. In practice this means an individual change score of a least four points equates to a medium effect size and significant change. This threshold is the same for all three HoNOS measures.
Computation:
(Numerator ÷ Denominator) x 100
Calculated separately for each group.
Numerator:
Number of completed or ongoing episodes of mental health care with completed outcome measures, partitioned by setting, where significant improvement/significant deterioration/no significant change was identified between baseline and follow-up within the reference period.
Numerator data elements:
Denominator:
Number of completed or ongoing episodes of mental health care with completed outcome measures, partitioned by mental health setting within the reference period.
The NOCC, maintained by the Department of Health compiles all state and territory consumer outcomes data on an annual data submission basis. However, NOCC does not allow linkage of episodes of care across financial years. This limitation does not exist for states and territories own data sets.
Compared to data constructed from state and territory datasets and using complete episodes, this limitation is likely to result in calculation of the indicator based on a significantly smaller pool of episodes. It may also have a systematic effect on apparent results on this indicator, but the direction and size of this effect has not been systematically tested.
Other issues caveats:
Change in consumers’ clinical outcomes is not presented for states and territories with less than 200 unique consumer outcomes identifiable within the reference period. States and territories excluded from individual publication are used to calculate the change in consumers’ clinical outcomes for Australia.
Greater variability in HoNOS scores can be expected, if effect sizes were calculated using ‘locally’ derived standard deviations. Statistically, fewer observations generally give rise to greater variability.
Source and reference attributes
Submitting organisation:
Australian Institute of Health and Welfare on behalf of the National Mental Health Performance Subcommittee
Reference documents:
Key Performance Indicators for Australian Public Mental Health Services, Third edition (2014)
