Academy of Social Sciences in Australia
Download this issue of Dialogue [PDF: Filesize
2.54 MB]
President's Report
Professor Barry McGaw
The articles in this edition of Dialogue and two recent events have provoked me to reflect on the role of statistics and indicators in my former work as Director for Education at the Organisation for Economic Policy and Development (OECD) and my current work as Chair of the Board of the Australian Curriculum, Assessment and Reporting Authority (ACARA).
One of the recent events was collection of Australia's 2011 Census. It provided a good reminder of how one of the nation's most important databases is maintained. The other was a small, high-level Workshop to Enhance the National Statistical System convened on 17-18 August 2011 by the COAG Reform Council and the Australian Statistics Advisory Council. It considered the needs of a broad range of policies, including economic, health, education and environment.
The collection and publication of internationally comparable statistics and indicators is a major emphasis in all of OECD's work. For a long period, the OECD's education statistics and indicators focused primarily on inputs including funds and a range of inputs that the funds provide such as teaching staff. Outcome measures were generally limited to participation and graduation rates through various levels of education and employment and unemployment rates by level of education.
In developing indicators it is important not to let the perfect be the enemy of the good but it is important to improve the validity and reliability of the indicators if they are to be put to serious use. Publishing with limited coverage, as with empty cells in international tables for countries unable or unwilling to provide the data typically leads quite rapidly to improved coverage. Whether to publish and use indicators that are in some ways inadequate is a less straightforward matter but, if the data satisfy reasonable validity and reliability requirements, it is often best to publish and then to improve.
Indicators of human capital provide a good example. The initial indicator was the number of years of formal education completed, called 'educational attainment'. It requires the assumption that a given number of years of education is equivalent regardless of the country or the type of institution in which the education was completed. In the late 1990s, the OECD resolved to move beyond this indicator by collecting information on the quality of education. Through the Program for International Student Assessment (PISA), the OECD is measuring the performances of 15-year-olds in reading, mathematics and science on a three-yearly cycle that began in 2000. The OECD's work built on national work in a number of member countries, including Australia, and on the work of an international non-governmental organisation, the International Association for the Evaluation of Educational Achievement (IEA), that had been collecting such data in a range of subject areas since 1959. (See www.iea.nl)
With the initial indicator, 'educational attainment', many investigations have shown the importance of human capital. OECD's analysis of factors that might account for the differences in rates of economic growth among OECD countries in the 1990s examined the effects of investment share, human capital (measured as educational attainment), population growth, variation of inflation, size of the government sector and trade exposure. Only changes in human capital and trade exposure accounted1 consistently for the differences in economic growth .
Investigations of the relationship between human capital and economic growth that use an indicator of the quality of education reveal a much stronger link than those found using educational attainment as the indicator. Hanushek and others have established this using international data on student2 3 4 achievement from the IEA and OECD/PISA surveys as the measure of quality
Moving from an indicator of quantity to an indicator of quality is just a first step. Quality measures can be improved. In the first OECD/PISA survey in 2000, reading was the main domain, with more students' time taken in assessments of it than either mathematics or science. In 2003, mathematics was the main domain for assessment and reading and science were minor domains. That provided the opportunity to conceptualise the mathematics performance domain more fully than for the 2000 survey, with the consequences that the 2003 science achievement scale cannot be linked to the 2000 scale and that the time series for mathematics now begins with the 2003 data. In a similar way, though science performance was assessed in 2000 and 2003, a new scale was introduced in 2006 when science was the major domain of assessment. The 2000 mathematics assessments and the 2000 and 2003 science assessments contributed to the transformation in international education policy discussion wrought by PISA 2000 but there was no sense that they should be preserved in aspic to achieve a longer time series when the opportunity came for the measures to be improved.
The OECD/PISA surveys also collect information on students' social backgrounds. In all countries there is a relationship between social background and educational achievement with more socially advantaged students generally performing at a higher level than less socially advantaged students. The 'social gradient' - the steepness of the regression line for this kind of relationship - is, however, not the same in all countries. Australia's social gradient is significantly steeper than that for the OECD5 as a whole while those for Finland, Canada, Korea and Japan are significantly less steep . Australia almost matches these countries in quality so Australia's challenge is not only to close that quality gap but also to match these others in equity by achieving a less steep social gradient.
One purpose of the My School website developed by the Australian Curriculum, Assessment and Reporting Authority is to provide a basis for reducing the impact of differences in students' social backgrounds on educational achievement in Australia. In My School, the current measures of quality of outcomes are scores on the National Assessment Program: Literacy and Numeracy (NAPLAN) tests. The measure of social background is provided by an index of community socio-educational advantage (ICSEA) which is based on parents' education and occupation. The relationship at the school level between ICSEA and NAPLAN results is shown in the figure below:
ICSEAs have been set with a mean of 1000 and a standard deviation of 100. The overall distribution is somewhat skewed. The scale shown in Figure 1 is a little truncated. There are 75 schools with ICSEAs in the range below 800, down to 548. They are mostly small, remote schools for indigenous students. The truncation in the display at 1200 excludes only 13 schools with ICSEAs from 1201 to 1233. The regression line shown in Figure 1 is for the full data set.
Schools above the regression line in Figure 1 can be said to have NAPLAN achievements better than could have been expected on the basis of their ICSEAs and so to have added more value than could have been expected. Those below the line could be said to have NAPLAN achievements worse than could have been expected on the basis of their ICSEAs and so to have added less value than could have been expected.
Deviations from the regression line are sometimes used as indicators of 'value-added' with the implication that those well above the regression line show how much might be achieved and can serve as models for improvement for those well below the regression line.
By that logic both schools B and C with positive deviations would be identified as schools from which school A with a negative deviation might learn. Basing an index on deviations alone obscures the fact that school B has students from a much more advantaged social background than the students in school A. Telling staff in school A that they might learn from school B invites the rejoinder that school A's superior achievements reflect its students' more advantaged social backgrounds, regardless of what the regression analysis might show. Comparison of school A with school C is much more telling and potentially helpful. The fact that school C achieves much better results than school A with students from similar social backgrounds denies school A the opportunity to claim that its performance is an inevitable consequence of the social backgrounds of its students. Comparison with school C is challenging but fair and opens the possibility of school A seeking to learn from school C about more effective policies and practices that it might implement. The My School website gives each school a comparison with the 30 schools immediately above and the 30 immediately below it on the ICSEA scale.
Central to the value of these comparisons is the validity of the claim that the 60 comparison schools do deal with students from similar backgrounds. For the first version of My School information on parents' education and occupation was obtained indirectly from Census data. Using the relatively small (around 250 households) and relatively homogeneous Census Collection Districts, the average education and occupation data for adults in a district were assigned to all students whose home address was in the district. This requires the assumption that students from the district who attend non-government schools and those who attend government schools have essentially the same social backgrounds. If they do not, the strategy falls foul of the 'ecological fallacy', which applies when the characteristics of groups may not appropriately be attributed to individual members of the group. For the second version of My School, direct information on the actual education and occupations of students' own parents was obtained for more than 90 per cent of students. Compared with the ICSEAs calculated from Census Collection District data, these direct data generated ICSEAs that were generally lower for government schools, generally higher for independent schools and generally marginally higher for Catholic schools. The differences clearly confirm that the ICSEAs based on Census Collection District data had fallen foul of the ecological fallacy.
The two ICSEA scales are highly correlated for the 9500 schools in Australia but the systematic differences in values for schools of different types support the use of the one based on direct parent data. As further evidence for its superiority, it accounts for about 78 per cent of the variance in schools' mean performances on NAPLAN while the ICSEA scale based on Census Collection District data accounts for around 68 per cent.
The change in the basis of derivation of ICSEA values meant that for each school the group of 60 other schools with students from similar social backgrounds was different on the second occasion from the one used on the first occasion. The groupings will be more stable in the future, altering only as the demography of students in the schools changes. This is another example of abandoning consistency over time for improvement.
While all of this points clearly to the advantage of using the ICSEA scale based on parent information, there are questions that need to be considered about the quality of the data obtained from parents. The data are obtained in self-reports from parents and recorded direct by parents or by school staff on the basis of interviews of parents. Census data are self-report data too, of course, but they are collected in different circumstances and may be more valid and reliable.
The data on parents are collected when students are first enrolled in school so all of the data used in My School to date have been collected before their use in creating ICSEA was even planned. Now that use is known, there is always a risk that there could be some gaming of the system in future data collections.
The solution might be to develop new ways of gathering the information from parents, for example having it done by a third-party agency independent of the school and of the Australian Curriculum, Assessment and Reporting Authority.
The need for change can also arise if the conditions of data collection threaten the validity of the data. Responses can be biased by knowledge of the planned use of the data. Responses can be shaped by the data collection itself. Asking individuals about their attitudes can alter their attitudes if they are provoked by the question to reflect in new ways on the issue behind the question. Webb, Campbell & Schwartz6 described such measures as 'reactive' and proposed the careful selection of non-reactive measures instead. In an earlier edition7 they described the non-reactive measures as unobtrusive and provided a wonderful series of examples of unobtrusive measures such as the rate of replacement of floor covering in front of museum exhibits as an indicator of popularity and a count of empty containers coming out the back door in the garbage rather than questions asked of residents at the front door to establish alcohol consumption.
In the development and use of indicators, the competing needs for consistency and improvement will always be in tension. Longer time series are valuable but so too is improved validity. Indicators and the statistics behind them tell only part of the story. They can represent a state of affairs. They can illustrate trends over time. They can suggest further questions to be explored but they cannot capture the richness of human behaviour and social interactions. Nor can they provide explanations of causes for correlations they might reveal. For that we need qualitative data as well and a rich array of social research methodologies.
Multidisciplinary collaboration is the best protection against the blinkered view of the world that can be imposed by one's preferred methodology. The Academy seeks to sponsor that within the social sciences and, through collaboration with the other Academies, on a broader scale as well.
Footnotes
- Organisation for Economic Co-operation and Development. (2000). OECD Economic Outlook, December 2000. Table A14.1.Paris, OECD.
- Hanushek, EA & Kimko, DD (2000). 'Schooling, labor-force quality, and the growth of nations', American Economic Review, 90, 1184-1208.
- Hanushek, EA & Wößmann, L (2007). The role of education quality in economic growth, World Bank Policy Research Working Paper 4122. World Bank, Washington.
- Organisation for Economic Co-operation and Development. (2010). The high cost of low educational performance: the long-run economic impact of improving PISA outcomes. Author, Paris.
- Organisation for Economic Co-operation and Development. (2007). PISA 2006: science competencies for tomorrow's world. Vol 1 - analysis. Author, Paris: p.184.
- Webb, EJ, Campbell, DT & Schwartz, RD (1981). Nonreactive measures in the social sciences. Houghton Mifflin, Dallas.
- Webb, EJ, Campbell, DT, Schwartz, RD & Sechrest, L (1966). Unobtrusive measures: nonreactive research in the social sciences. Rand McNally, Chicago.
Barry McGaw
2011
Professor Barry McGaw was elected president of the Academy at the AGM in November. He is half-time Professorial Fellow at the Melbourne Graduate School of Education at the University of Melbourne where he is Executive Director of the Cisco-Intel-Microsoft Assessment and Teaching of 21st Century Skills project. He is Chair of the Australian Curriculum, Assessment and Reporting Authority.
Last modified: 3 October 2011.