Australian Government: Australian Institute of Health and Welfare METeOR Home Page

Online Services Report (OSR) Database, 2016–17; Quality Statement

Identifying and definitional attributes

Metadata item type:Help on this termQuality Statement
Synonymous names:Help on this termOATSIH Services Reporting
METeOR identifier:Help on this term688427
Registration status:Help on this termAIHW Data Quality Statements, Endorsed 10/07/2018

Data quality

Quality statement summary:Help on this term

The Online Services Report (OSR) annually collects information from organisations that receive Australian Government funding to provide health services to Aboriginal and Torres Strait Islander people. The main data quality issues identified in the 2016–17 collection were incomplete reporting and data discrepancies between two or more questions. As part of the collection, AIHW staff contacted organisations to follow up data queries and to request revised data where necessary.

In 2016–17, 266 (97% of in scope) organisations contributed to the OSR national report. Most organisations (94%) had all of their data included in national analyses; 6% (16 organisations) had 1 or more data items excluded from national analyses due to data quality issues.

While the collection and validation processes were similar to previous collections, there were some changes to the 2016–17 collection that are worth noting. The first were some changes to the types of contacts included in the episodes of care count, which meant some contact types previously included in the episode of care count in some organisations were now excluded from the count. There was also a change in how some data was extracted and pre-populated onto OCHREStreams. A new direct load process developed by the Department of Health (DoH) was introduced and was used to extract some data directly from an organisation's clinical information system (CIS) onto OCHREStreams, the online portal for OSR submissions. This direct load process replaced a previous extraction tool used across the different systems. Data that could be directly loaded include client numbers, episodes of care, client contacts, adult and child health checks, chronic disease management plans, and antenatal clients and visits. This involved upgrades to the software being used by some organisations. 

Investigation of 2016—17 OSR data suggests most of the data should be comparable with previous years and can be reported on nationally. This is because the questionnaire has not changed and, as in previous collections, most data entry was done manually. However, some changes to the direct load data were noted. These include:

  1. The number of episodes of care reported in 2016–17 was lower and this represents a break in time series compared with previous collections.
  2. The relationship between episodes of care and contact counts is not as close as in previous collections.
  3. There were large increases in health check and chronic disease management plan data in some organisations.

The mapping of these items need further checking, both across the different types of CIS and against the OSR definition. Given this, the current national report does not report on episodes of care time series or on health checks/chronic disease management plans (see Chapter 2 in the 2016—17 report for more information).

Institutional environment:Help on this term

The Australian Institute of Health and Welfare (AIHW) is responsible for the data management, analysis and reporting of information collected as part of the OSR. It downloads the OSR data submitted by health organisations on OCHREStreams, a secure web portal, hosted and managed by the Improvement Foundation on behalf of the Australian Government DoH. The Improvement Foundation (Australia) Ltd was established in Adelaide in 2006 as a not-for-profit organisation and a subsidiary of the UK Improvement Foundation. The Improvement Foundation provides expertise in the development and delivery of quality improvement techniques through specialist change management advice and IT systems.

The AIHW is a major national agency set up by the Australian Government under the Australian Institute of Health and Welfare Act 1987 to provide reliable, regular and relevant information and statistics on Australia's health and welfare. It is an independent corporate Commonwealth entity established in 1987, governed by a management Board, and accountable to the Australian Parliament through the Health portfolio.

The AIHW aims to improve the health and wellbeing of Australians through better health and welfare information and statistics. It collects and reports information on a wide range of topics and issues, ranging from health and welfare expenditure, hospitals, disease and injury, and mental health, to ageing, homelessness, disability and child protection.

The Institute also plays a role in developing and maintaining national metadata standards. This work contributes to improving the quality and consistency of national health and welfare statistics. The Institute works closely with governments and non-government organisations to achieve greater adherence to these standards in administrative data collections to promote national consistency and comparability of data and reporting.

One of the main functions of the AIHW is to work with the states and territories to improve the quality of administrative data and, where possible, to compile national datasets based on data from each jurisdiction, to analyse these datasets and disseminate information and statistics.

The Australian Institute of Health and Welfare Act 1987, in conjunction with compliance to the Privacy Act 1988, (Cth) ensures that the data collections managed by the AIHW are kept securely and under the strictest conditions with respect to privacy and confidentiality.

For further information see the AIHW website www.aihw.gov.au.

Timeliness:Help on this term

The OSR collects data for each financial year and has been ongoing since 2008–09. For most years, including 2016–17, the reporting period was for the financial year from 1 July to 30 June. In 2014–15, the Census date for the collection was brought forward by 1 month to 31 May 2015, rather than 30 June 2015. This was to allow organisations to use the PenCAT data extraction tool, which would not be available for OSR reporting after 30 June 2015. Organisations were still asked to provide data for a 12 month period (1 June 2014 to 31 May 2015) to enable time series analyses. However, this meant data for June 2014 may have been reported in both the 2013–14 and 2014–15 collections for clients, contacts and episodes of care. Nevertheless, comparing 2013–14, 2014–15 and 2015–16 data did not show any obvious underreporting for any data items at the national level.

In 2016–17, 275 organisations were in-scope for the OSR collection and were required to submit initial data by 31 August which is within two months of the census date (30 June). Sixty percent of organisations had submitted data by this date. Some organisations were given an extension by the DoH until the 15 September and by this date 94% had done their first submission. By the final cut-off date, 266 organisations (97%) had submitted data and 9 (3%) organisations had not submitted and were non-response.

Accessibility:Help on this term

National level data are available in AIHW publications. The most current report is the "Aboriginal and Torres Strait Islander health organisations, Online Services Report-key results 2016–17". This publication is available in hard copy or on the AIHW website.

Interpretability:Help on this term

Data were collected using a web-based reporting tool designed to capture information on the health services provided to Aboriginal and Torres Strait Islander clients attending primary health, New Directions mothers and babies services, substance-use, and social and emotional wellbeing services funded by the Australian Government. Data for a small number of non-Indigenous clients attending these services were also included. The statistical analyses used in reports are mainly descriptive. Numbers and percentages are commonly used to show major trends. The annual publication also includes a range of information to assist in the interpretation of data, including a glossary and information on data quality.

A characteristic of the collection is that the number of organisations submitting data changes slightly each year. While for the most part, it is the same organisations contributing to the collection, the number of organisations may change due to changes in funding, auspicing or reporting arrangements at the local level. In 2016–17, 266 organisations contributed to the collection. This was a decrease of 11 from 2015–16 (277 organisations). There were 13 organisations reporting in 2015–16 that did not report in 2016–17 (5 were no longer in scope and 8 were non-response), while 2 organisations not reporting in 2015–16 were in scope for reporting in 2016–17.

Another characteristic of the collection is that the organisations submitting valid data for a particular data item may also change over time, as data with remaining quality issues after follow-up are excluded from national analyses. This means that each year some organisations' data are partially accepted rather than fully accepted. Both the changing organisations in-scope for the collection and the changing number of organisations with valid data for each data item may impact time series analyses, as this is based on all the organisations that provide valid data in each year, rather than on a subset which have valid data for all years.

Relevance:Help on this term

Data are available on funded organisations from 9 reporting periods from 2008–09 to 2016–17. It provides an overview of Australian Government-funded health organisations that aim to improve health outcomes for Aboriginal and Torres Strait Islander people. In 2016–17, most OSR organisations (74%) provided primary health-care services, either solely or in combination with the New Directions mothers and babies program, social and emotional wellbeing services and substance-use rehabilitation and treatment services.

Data presented in the national report are reported at an aggregated level. There are chapters looking at primary health care, maternal and child health, social and emotional wellbeing and substance-use. This aggregated analysis provides general information about services offered at primary health and other organisations, for example clinical care, health promotion and group activities as well as client numbers, contacts, episodes of care and full time equivalent staffing levels. The data can also be analysed at jurisdictional and remoteness area level. Most of the organisations that report to the OSR also report to the national key performance indicators (nKPI) collection. OSR data therefore provides useful contextual information for organisations participating in the nKPIs.

There may be questions that are not relevant to all organisations. There may also be instances where an organisation's data are not included for a question due to data quality concerns. The denominators used for percentage calculations are the number of organisations that provided valid data for each question.

Accuracy:Help on this term

While the collection and validation processes were similar to previous collections, there were changes to the 2016–17 collection that are worth noting. The first was a change to the types of contacts included in the episodes of care count, which meant some contact types (for example, health care delivered over the telephone and hospital-related contacts) previously included in the episode of care count in some organisations were now excluded from the count. There was also a push to ensure the episode of care definition was adhered to i.e. no more than 1 episode of care was counted for a client in a calendar day, across all types of CIS. These changes were made to address perceived inconsistencies in reporting between organisations using different systems and to improve consistency in the application of the counting rules for the collection. There was also a change in how data was extracted and pre-populated onto OCHREStreams. A new direct load process developed by the DoH was introduced across the different types of CIS. This meant some OSR data (client numbers, episodes of care, client contacts, adult and child health checks, chronic disease management plans, and antenatal clients and visits) could be extracted from the CIS used by the organisation directly onto OCHREStreams, a web-based portal built and maintained by the Improvement Foundation for the DoH. As in previous collections however, organisations could still submit their data manually, overwrite and make changes to their data on OCHREStreams (for example if they assessed it to be inaccurate), and have all submissions approved by their CEO, before the data were made available to the AIHW for analysis.

The AIHW examined all OSR submissions to identify missing data and possible data quality issues and in 2016-17 had at least one query for 258 (97%) organisations. AIHW staff contacted these organisations for further clarification and review and around three-quarters of organisations (74%) resubmitted some data. Some organisations were unable to provide exact data and provided estimates of client numbers and episodes of care. In 2016-17, 20 (10%) organisations providing primary health-care services estimated their episodes of care and 31 (16%) estimated their client numbers. This data was accepted unless the AIHW assessed their data was likely to overestimate actual numbers.

Common data quality queries in the 2016-17 collection were around incomplete or inaccurate data (for example, workforce positions were not reported or were reported in terms of the number of people rather than full-time equivalent positions); data discrepancies between two or more questions (for example, the number of clients exceeded the number of episodes of care); and large increases or decreases in data compared with previous submissions. Where significant data quality issues remained after follow-up then these data were excluded from national analyses.

In 2016-17, by the final cut-off date for submissions, most organisations (94%) had provided data that could be included in national analyses. The remaining 6% (16 organisations) had a total of 32 data items excluded from national analyses due to remaining data quality issues (some organisations had more than one item excluded). Exclusion rates vary by data item. For example, primary health episodes of care data were excluded for 2 (1%) primary health-care organisations, while substance-use client data were excluded for 5 (6%) organisations that provided this service.

Investigation of 2016—17 data suggests that while the level of non-response was higher, most of the data should be comparable with previous years and be able to be reported on nationally. This is because the questionnaire has not changed and, as in previous collections, most data entry was done manually. Of the approximately 130 questions in the OSR, only around 5 to 9 questions (depending on the type of CIS used) could be pre-populated using the new direct load process. The remaining questions were tick-box questions (for example the types of programs/activities an organisation provides), open-text/comments, or questions that require numeric data that could not be pre-populated (e.g. workforce numbers).

The impacts of moving to direct load for the pre-populated data are still being investigated. While logic failures are relatively straightforward to identify and can usually be addressed by an organisation, time trend issues are more difficult for organisations to validate because of normal fluctuations in the data over time. In 2016-17, the change in scope of the episodes of care data and move to direct load further obscured time trend issues, and organisations were sometimes not able to advise whether changes were due to ‘real world’ changes or a result of the new extraction process. In some cases however, these changes have had an impact on data comparability, as outlined below.

  1. The change in scope and mapping of episodes of care data meant a decrease in episodes of care was expected, in particular in organisations using Communicare. However there was an unexpected decrease in episode counts in organisations using Medical Director. Some organisations had a recording/extract issue that could not be resolved by the end of the collection. Some contacts were not recorded in a way that could be extracted as part of their OSR direct load, which meant contacts for some health workers were not included as part of their episodes of care count as they should have been. This meant that in addition to the changes in mapping for episodes of care, this data were underestimated in some organisations. The overall number of episodes of care reported in 2016–17 was 16% lower than the number reported in 2015–16, and this represents a break in time series compared with previous collections. Episodes of care time series are therefore not presented in the 2016–17 national report.
  2. Compared with 2015–16, in some organisations there were large decreases in episodes of care and increases in client contacts (excluding transport contacts). In previous collections there was a closer relationship between the type of contacts that would count as an episode of care and a client contact (excluding transport). In 2016–17 the ratio of contacts to episodes increased and the relationship between the two counts is not as close.
  3. There were large increases in health checks and chronic disease management plans reported in some organisations using Medical Director and Communicare. Further investigation of how the mapping was done for these items is needed to determine whether they are comparable with previous collections and if they were collected consistently across all systems. Given this, child health check data have been excluded from the national report this year.
Coherence:Help on this term

Data were collected and reported by the AIHW for the first time in 2008–09 using a new paper-based OSR questionnaire. This replaced the Service Activity Reporting, Drug and Alcohol Services Reporting, and Bringing Them Home and Link Up counselling data collections. These were previously collected by the DoH. It would not be appropriate to make time series comparisons with data collected prior to 2008–09, due to differences in the collection methodology and scope of the collection.

In 2012–13, following a review of the OSR collection, a revised collection instrument was introduced that subdivided the questionnaire into modules for ease of completion. The new form also introduced a reporting module for maternal and child health care programs and activities. Major questions (e.g. episodes of care, clients and contacts) remained the same. Some other questions were revised or restructured. For example, the question on clinical and health related activities/programs from the previous collections was split into several new questions and some answer options were no longer available. Only comparable data were included in time series analyses in the national report. Additional questions on the gaps and challenges faced by organisations in providing health services were included in the questionnaire and analysed for the first time. Another change to the form in 2012-13 meant some information was now available at the site level and not just the organisation level, including the address of each service delivery site and the range of services provided from each site. This information can be used to more accurately identify service delivery locations and for data validation. For example, using this information in 2015-16 meant the remoteness category was changed for a few organisations to better reflect service delivery rather than their administrative address.

From 2012–13, the collection was also assisted by the introduction of the PenCAT audit tool and the OCHREStreams online reporting portal, developed and maintained by the Improvement Foundation for the DoH. The PenCAT tool could be used to extract some data (including counts of clients, client contacts and episodes of care) directly from an organisation’s CIS onto OCHREStreams. The AIHW could then download this data from the web portal. These changes were designed to improve the accuracy of the data and reporting response times and to reduce respondent burden, however, it may have led to some differences in the data reported for some organisations. It is important to note that organisations could still overwrite this pre-population function and not all organisations used this facility, so many still submitted all of their data manually onto the web portal. The PenCAT tool was not available for use for extracting data onto OCHREStreams from 1 July 2015 and was not used for the 2015–16 collection. For this collection organisations relied on their own data extraction processes and manually entered this data onto OCHREStreams.

The way in which pre-populated data was extracted onto OCHREStreams changed again for the 2016–17 collection. The DoH worked with CIS vendors and introduced direct load to extract this data directly from an organisations CIS onto OCHREStreams. As in previous collections however, some organisations still submitted their OSR data manually and all organisations were able to overwrite and make changes to their data before submitting them to the AIHW.

New Directions: Mothers and Babies services were in-scope for the OSR for the first time in the 2013–14 collection. It should be noted that the number of organisations receiving New Directions funding that submitted OSR data are different to the number of New Directions funded organisations reported by the DoH, due to differences in how an organisation is defined. An organisation defined as a single entity for funding purposes may have multiple organisations reporting OSR data. Most of the organisations receiving New Directions funding were already contributing to the OSR collection, because they also received primary health-care funding. 

Data products

Implementation start date:Help on this term01/12/2017

Source and reference attributes

Submitting organisation:Help on this term

AIHW

Relational attributes

Related metadata references:Help on this term

Supersedes Online Services Report (OSR) Database, 2015–16; Quality Statement AIHW Data Quality Statements, Archived 10/07/2018

My items Help on this term
Download Help on this term