Международное исследование PISA: оценка образовательных систем ОЭСР

Scholastic performance study by the OECD

The Programme for International Student Assessment (PISA) is a worldwide study by the Organisation for Economic Co operation and Development (OECD) in member and non member nations intended to evaluate educational systems by measuring 15 year old school pupils' scholastic performance on mathematics, science, and reading. It was first performed in 2000 and then repeated every three years. Its aim is to provide comparable data with a view to enabling countries to improve their education policies and outcomes. It measures problem solving and cognition. The results of the 2022 data collection were released in December 2023.

PISA, and similar international standardised assessments of educational attainment are increasingly used in the process of education policymaking at both national and international levels. PISA was conceived to set in a wider context the information provided by national monitoring of education system performance through regular assessments within a common, internationally agreed framework; by investigating relationships between student learning and other factors they can "offer insights into sources of variation in performances within and between countries". Until the 1990s, few European countries used national tests. In the 1990s, ten countries / regions introduced standardised assessment, and since the early 2000s, ten more followed suit. By 2009, only five European education systems had no national student assessments.

Data from international standardised assessments can be useful in research on causal factors within or across education systems. They allow typologies to be developed that can be used for comparative statistical analyses of education performance indicators, thereby identifying the consequences of different policy choices. They have generated new knowledge about education: PISA findings have challenged deeply embedded educational practices, such as the early tracking of students into vocational or academic pathways. 79 countries and economies participated in the 2018 data collection. Barroso and de Carvalho find that PISA provides a common reference connecting academic research in education and the political realm of public policy, operating as a mediator between different strands of knowledge from the realm of education and public policy. However, although the key findings from comparative assessments are widely shared in the research community

Policy makers in most participating countries see PISA as an important indicator of system performance; PISA reports can define policy problems and set the agenda for national policy debate; policymakers seem to accept PISA as a valid and reliable instrument for internationally benchmarking system performance and changes over time; most countries—irrespective of whether they performed above, at, or below the average PISA score—have begun policy reforms in response to PISA reports. In Hungary, by comparison, which shared similar conditions to Germany, PISA results have not led to significant changes in educational policy. Because many countries have set national performance targets based on their relative rank or absolute PISA score, PISA assessments have increased the influence of their (non elected) commissioning body, the OECD, as an international education monitor and policy actor, which implies an important degree of 'policy transfer' from the international to the national level; PISA in particular is having "an influential normative effect on the direction of national education policies". PISA data can be "used to fuel long standing debates around pre existing conflicts or rivalries between different policy options, such as in the French Community of Belgium". In such instances, PISA assessment data are used selectively: in public discourse governments often only use superficial features of PISA surveys such as country rankings and not the more detailed analyses. Rey (2010:145, citing Greger, 2008) notes that often the real results of PISA assessments are ignored as policymakers selectively refer to data in order to legitimise policies introduced for other reasons. In addition, PISA's international comparisons can be used to justify reforms with which the data themselves have no connection; in Portugal, for example, PISA data were used to justify new arrangements for teacher assessment (based on inferences that were not justified by the assessments and data themselves); they also fed the government's discourse about the issue of pupils repeating a year, (which, according to research, fails to improve student results). In Finland, the country's PISA results (that are in other countries deemed to be excellent) were used by Ministers to promote new policies for 'gifted' students. Such uses and interpretations often assume causal relationships that cannot legitimately be based upon PISA data which would normally require fuller investigation through qualitative in depth studies and longitudinal surveys based on mixed quantitative and qualitative methods, which politicians are often reluctant to fund. Recent decades have witnessed an expansion in the uses of PISA and similar assessments, from assessing students' learning, to connecting "the educational realm (their traditional remit) with the political realm". This raises the question of whether PISA data are sufficiently robust to bear the weight of the major policy decisions that are being based upon them, for, according to Breakspear, PISA data have "come to increasingly shape, define and evaluate the key goals of the national / federal education system". The PISA mathematics literacy test asks students to apply their mathematical knowledge to solve problems set in real world contexts. To solve the problems students must activate a number of mathematical competencies as well as a broad range of mathematical content knowledge. TIMSS, on the other hand, measures more traditional classroom content such as an understanding of fractions and decimals and the relationship between them (curriculum attainment). PISA claims to measure education's application to real life problems and lifelong learning (workforce knowledge). In the reading test, "OECD/PISA does not measure the extent to which 15 year old students are fluent readers or how competent they are at word recognition tasks or spelling." Instead, they should be able to "construct, extend and reflect on the meaning of what they have read across a wide range of continuous and non continuous texts." PISA also assesses students in innovative domains. In 2012 and 2015 in addition to reading, mathematics and science, they were tested in collaborative problem solving. In 2018 the additional innovative domain was global competence.

PISA is sponsored, governed, and coordinated by the OECD, but paid for by participating countries.

The students tested by PISA are aged between 15 years and 3 months and 16 years and 2 months at the beginning of the assessment period. The school year pupils are in is not taken into consideration. Only students at school are tested, not home schoolers. In PISA 2006, however, several countries also used a grade based sample of students. This made it possible to study how age and school year interact. To fulfill OECD requirements, each country must draw a sample of at least 5,000 students. In small countries like Iceland and Luxembourg, where there are fewer than 5,000 students per year, an entire age cohort is tested. Some countries used much larger samples than required to allow comparisons between regions.

Each student takes a two hour computer based test. Part of the test is multiple choice and part involves fuller answers. There are six and a half hours of assessment material, but each student is not tested on all the parts. Following the cognitive test, participating students spend nearly one more hour answering a questionnaire on their background including learning habits, motivation, and family. School directors fill in a questionnaire describing school demographics, funding, etc. In 2012 the participants were, for the first time in the history of large scale testing and assessments, offered a new type of problem, i. e. interactive (complex) problems requiring exploration of a novel virtual device. In selected countries, PISA started experimentation with computer adaptive testing.

Countries are allowed to combine PISA with complementary national tests. Germany does this in a very extensive way: On the day following the international test, students take a national test called PISA E (E=Ergänzung=complement). Test items of PISA E are closer to TIMSS than to PISA. While only about 5,000 German students participate in the international and the national test, another 45,000 take the national test only. This large sample is needed to allow an analysis by federal states. Following a clash about the interpretation of 2006 results, the OECD warned Germany that it might withdraw the right to use the "PISA" label for national tests.

From the beginning, PISA has been designed with one particular method of data analysis in mind. Since students work on different test booklets, raw scores must be 'scaled' to allow meaningful comparisons. Scores are thus scaled so that the OECD average in each domain (mathematics, reading and science) is 500 and the standard deviation is 100. This is true only for the initial PISA cycle when the scale was first introduced, though, subsequent cycles are linked to the previous cycles through IRT scale linking methods. This generation of proficiency estimates is done using a latent regression extension of the Rasch model, a model of item response theory (IRT), also known as conditioning model or population model. The proficiency estimates are provided in the form of so called plausible values, which allow unbiased estimates of differences between groups. The latent regression, together with the use of a Gaussian prior probability distribution of student competencies allows estimation of the proficiency distributions of groups of participating students. The scaling and conditioning procedures are described in nearly identical terms in the Technical Reports of PISA 2000, 2003, 2006. NAEP and TIMSS use similar scaling methods.

All PISA results are tabulated by country; recent PISA cycles have separate provincial or regional results for some countries. Most public attention concentrates on just one outcome: the mean scores of countries and their rankings of countries against one another. In the official reports, however, country by country rankings are given not as simple league tables but as cross tables indicating for each pair of countries whether or not mean score differences are statistically significant (unlikely to be due to random fluctuations in student sampling or in item functioning). In favorable cases, a difference of 9 points is sufficient to be considered significant. PISA never combines mathematics, science and reading domain scores into an overall score. However, commentators have sometimes combined test results from all three domains into an overall country ranking. Such meta analysis is not endorsed by the OECD, although official summaries sometimes use scores from a testing cycle's principal domain as a proxy for overall student ability.

Period Focus OECD countries Partner countries Participating students Notes 2000 Reading 28 4 + 11 265,000 The Netherlands disqualified from data analysis. 11 additional non OECD countries took the test in 2002. 2003 Mathematics 30 11 275,000 UK disqualified from data analysis, due to its low response rate. Also included test in problem solving. 2006 Science 30 27 400,000 Reading scores for US disqualified from analysis due to misprint in testing materials. 2009 Reading 34 41 + 10 470,000 10 additional non OECD countries took the test in 2010. 2012 Mathematics 34 31 510,000

Finland, which received several top positions in the first tests, fell in all three subjects in 2012, but remained the best performing country overall in Europe, achieving their best result in science with 545 points (5th) and worst in mathematics with 519 (12th) in which the country was outperformed by four other European countries. The drop in mathematics was 25 points since 2003, the last time mathematics was the focus of the tests. For the first time Finnish girls outperformed boys in mathematics narrowly. It was also the first time pupils in Finnish speaking schools did not perform better than pupils in Swedish speaking schools. Former minister of Education and Science Krista Kiuru expressed concern for the overall drop, as well as the fact that the number of low performers had increased from 7% to 12%.

India participated in the 2009 round of testing but pulled out of the 2012 PISA testing, with the Indian government attributing its action to the unfairness of PISA testing to Indian students. India had ranked 72nd out of 73 countries tested in 2009. The Indian Express reported, "The ministry (of education) has concluded that there was a socio cultural disconnect between the questions and Indian students. The ministry will write to the OECD and drive home the need to factor in India's "socio cultural milieu". India's participation in the next PISA cycle will hinge on this". The Indian Express also noted that "Considering that over 70 nations participate in PISA, it is uncertain whether an exception would be made for India". India did not participate in the 2012, 2015 and 2018 PISA rounds. A Kendriya Vidyalaya Sangathan (KVS) committee as well as a group of secretaries on education constituted by the Prime Minister of India Narendra Modi recommended that India should participate in PISA. Accordingly, in February 2017, the Ministry of Human Resource Development under Prakash Javadekar decided to end the boycott and participate in PISA from 2020. To address the socio cultural disconnect between the test questions and students, it was reported that the OECD will update some questions. For example, the word avocado in a question may be replaced with a more popular Indian fruit such as mango.

In 2015, the results from Malaysia were found by the OECD to have not met the maximum response rate. Opposition politician Ong Kian Ming said the education ministry tried to oversample high performing students in rich schools.

Sweden's result dropped in all three subjects in the 2012 test, which was a continuation of a trend from 2006 and 2009. It saw the sharpest fall in mathematics performance with a drop in score from 509 in 2003 to 478 in 2012. The score in reading showed a drop from 516 in 2000 to 483 in 2012. The country performed below the OECD average in all three subjects. The leader of the opposition, Social Democrat Stefan Löfven, described the situation as a national crisis. Along with the party's spokesperson on education, Ibrahim Baylan, he pointed to the downward trend in reading as most severe. According to an article of Sveriges Radio, poor immigrant children's scores are a significant cause of the recent decrease in Swedish Pisa scores.

Although PISA and TIMSS officials and researchers themselves generally refrain from hypothesizing about the large and stable differences in student achievement between countries, since 2000, literature on the differences in PISA and TIMSS results and their possible causes has emerged. Data from PISA have furnished several researchers, notably Eric Hanushek, Ludger Wößmann, Heiner Rindermann, and Stephen J. Ceci, with material for books and articles about the relationship between student achievement and economic development, democratization, and health; as well as the roles of such single educational factors as high stakes exams, the presence or absence of private schools and the effects and timing of ability tracking.

David Spiegelhalter of Cambridge wrote: "Pisa does present the uncertainty in the scores and ranks for example the United Kingdom rank in the 65 countries is said to be between 23 and 31. It's unwise for countries to base education policy on their Pisa results, as Germany, Norway and Denmark did after doing badly in 2001." According to a Forbes opinion article, some countries such as China, Hong Kong, Macau, and Argentina select PISA samples from only the best educated areas or from their top performing students, slanting the results. According to an open letter to Andreas Schleicher, director of PISA, various academics and educators argued that "OECD and Pisa tests are damaging education worldwide". According to O Estado de São Paulo, Brazil shows a great disparity when classifying the results between public and private schools, where public schools would rank worse than Peru, while private schools would rank better than Finland.