1. INTRODUCTION
Scientific literacy is an important part of
national quality and the foundation of social civilization and progress.
Scientific literacy is not only related to the comprehensive national strength
of the country, but also affects the quality of life of citizens in modern
society and subtly changing the public's values and views on problems. Without
the general improvement of the scientific quality of the whole people, it is
difficult to establish human resources of high-quality innovation and to
realize the rapid transformation of scientific and technological
achievements. Nowadays, many countries
have implemented promotion action on science literacy. For example, the State
Council of China formulated the national scientific literacy action plan
outline (2021-2035), and pointed out that scientific literacy of citizens
refers to advocating scientific spirit, establishing the scientific thought,
mastering the basic scientific method, understanding the necessary knowledge of
science and technology, and the application of its judgment and ability to
solve practical problems (The State Council
of China, 2021). Whether it is improving national scientific literacy or
understanding the implementation of science education, it is necessary to
evaluate the quality of science education. Through the comparison of the
content of the international mature scientific literacy evaluation project, it
has positive implications for promoting the development of science education,
improving the quality of science education in primary and secondary schools,
and cultivating excellent scientific and technological talents. Therefore, this
paper compared the scientific literacy evaluation projects of NAEP2019,
PISA2025 and NAEP2019 from the following four aspects: (1) development
progress, (2) definition of scientific literacy, (3) evaluation
concept and purpose, and (4) evaluation framework and contents, and then put forward
the enlightenment for China science education.
2.
DEVELOPMENT BACKGROUND OF NAEP, PISA AND TIMSS
In the 1960s and 1970s, in response to the
lack of information on academic performance in the United States, Francis
Keppel called for a national academic performance evaluation system and invited
renowned psychologist and educator Ralph W. Tyler to participate. After the
improvement of the assessment system in the 1980s and the emergence of the
National Assessment Regulatory Council of the United States, up to the 1990s
the National Assessment of Educational Progress (NAEP) model was fully
determined (Zhou,2005). NAEP is an evaluation program authorized by the US
Congress and administered by the National Center of Education Statistics within
the Institute of Educational Sciences to assess what American students know and
can do in various disciplines across the country, states and certain urban
areas. Since 1969, the evaluation has been implemented once a year, with 4, 8,
and 12 grade students. The evaluation includes nine subjects: reading,
mathematics, science, writing, American history, economics, citizenship,
geography, and art, but only two or three of them were selected in each round.
The latest NAEP2019 scientific evaluation framework mainly includes: Chapter 1
(overview); Chapter 2 (scientific content); Chapter 3 (scientific practice),
and Chapter 4 (Overview of evaluation design) (Deng & Yu,2022)..
In the early
1980s, with the mediocre trend sweeping the United States, the American
government, educational elites, employers and social parents queried the
quality of basic education in the United States. As a result, the United States
began to reflect on the quality of basic education. In addition, human beings
are faced with the challenges brought about by "knowledge society",
"information society" and "economic globalization", which
makes it urgent to establish an international learning quality monitoring
system and promote educational reform in various countries and improve the
quality of basic education (Zhang et al.,2011). Therefore, the Organization for
Economic Cooperation and Development (OECD) launched an International Student
Assessment Survey (PISA) in 1997. The project was planned in 2000 and conduct
tests every three years. PISA has conducted seven scientific literacy
assessments so far. The test subjects of PISA are students aged 15-16 who must
have received school education rather than family education, aiming to
understand whether students who have completed compulsory education have the
ability to adapt to future life (Liu &
Liu 2011). In 2023, the OECD released the Strategic Vision and Direction for
Science of PISA2025, which PISA is expected to evaluate in three dimensions:
scientific knowledge, scientific ability and scientific identity. Science
situation is no longer a dimension of PISA 2025 assessment, but implicit in the
dimension of scientific knowledge, scientific ability and scientific identity.
At the end
of the 20th century, the development of information and communication
technology promoted the communication and cooperation among countries in the
economic and cultural fields, which provided convenient conditions for
international organizations to carry out international cooperative research in
the field of education. Under the new situation and conditions, the new
situation has also appeared in the world comparative study (Lu, 2007). The International Mathematics
and Science Trends Research (TIMSS), which was initiated and organized by the
International Association for the Evaluation of Educational Achievements ( IEA)
to conduct evaluation research and evaluation activities of international
educational. Since 1995, an
international assessment of mathematics and science will be conducted among
fourth and eighth grade every four years. To date, TIMSS has completed its
seventh project evaluation (Gu, 2023). In 2023, the
IEA and the TIMSS & PIRLS International Research Center will evaluate
student trends in math and science.
3
DEFINITION OF SCIENTIFIC LITERACY
3.1 Definition
of Scientific Literacy in NAEP2019
NEAP2019 Point out that scientific literacy
is a fundamental goal for all young people in America. Through science
education, children begin to understand the world they live in and learn many
scientific principles about life. Furthermore, NEAP believes that the state has
an obligation to provide a solid foundation of post-secondary learning and work
experience for young people who choose to work in science and technology. NAEP
believes that a person with scientific literacy has the following four qualities:
first, be familiar with nature and understand the key facts, concepts,
principles and theories of science; second, link viewpoints from different
disciplines; third, use scientific principles and ways of thinking to improve
the understanding of the natural world; finally, use science to solve problems
in the real world. NAEP2019 has conducted an assessment from scientific content
and scientific practice (National
Assessment Governing Board, 2019).
3.2
Definition of Scientific Literacy in PISA2025
In 2015, PISA defined scientific literacy as
"the ability of a reflective citizen to use scientific thinking in
relevant scientific issues", believing that individuals with good
scientific literacy should have the following three abilities: scientifically
explain phenomena, design and evaluate scientific inquiry, and scientific
interpretation of data and evidence (OECD,2016).
PISA2025 points out that with the progress of human society, the world will
face many crises, which arises from the rapid development of science and
technology as well as the entry of the Earth into the Anthropocene
and may face more unknown challenges in the future. Humans may face more
challenges in the future. Therefore, PISA 2025 will pay more attention to the
interdisciplinary integration of scientific knowledge, the cultivation of
students' decision-making ability and the construction of scientific identity.
The new adjustment of this framework has a strong forward-looking and the
purpose of serving the reality.
3.3
Definition of Scientific Literacy in TIMSS2023
TIMSS2023 does not specify a general concept
of scientific literacy, but notes on its website that it expects young students
to become knowledgeable and scientifically literate citizens, who can
distinguish between scientific facts and fiction, and have a scientific basis
for understanding important social, economic and environmental issues. And in
the future, it can meet the needs of students for advanced learning in the
fields of science, technology and engineering to promote the solution of global
problems. TIMSS2023 will be evaluated from three aspects: scientific content,
scientific cognition and scientific practice (IEA, 2023).
3.4
Comparison of Definition of Scientific Literacy among NAEP2019, TIMSS2023, and
PISA2025
From the perspective of the differences of
scientific literacy, PISA focuses on the ability of citizens in scientific
inquiry; TIMSS emphasizes the ability of citizens in solving global problems;
NAEP not only requires citizens to have relevant scientific knowledge and
ability, but also to understand the relationship between science and society.
From the perspective of the commonality of the connotation of scientific
literacy, the three major evaluation projects are all started from the
individual level, requiring citizens to have certain scientific knowledge and
scientific ability to solve the social problems related to science.
4.
EVALUATION CONCEPT AND PURPOSE
4.1 The
Evaluation Concept and Purpose of NAEP2019
NAEP is the only national, representative and
continuous program that evaluates student academic achievement. Its scientific
literacy assessment is based on new trends in the National Science Education
Standards and Benchmarks for Scientific Literacy and some international
assessments such as PISA and TIMSS. The purpose of the evaluation is what the
students know and what they can do. It can be seen that NAEP emphasizes
practice, focusing on evaluating what they know and can do among American
primary and middle school students in various subjects. Evaluating and
analyzing the trend of students' academic achievement can provide data support
for educational decisions in the United States.
4.2 The
Evaluation Concept and Purpose of PISA2025
PISA scientific literacy assessment based on
the Educational Effectiveness Research theory and the theory of the latest
dynamic model was proposed by Creemers and Kyriakides. It is believed that educational efficiency is a
fusion of different research fields such as teacher behavior, curriculum,
organizational process, school management and educational policy. This fusion
is composed of direct or indirect factors that affect students' output
(including cognitive output and non-cognitive output). What’s more, factors at
different levels will interact with each other (Creemers & Kyriakides, 2008). The
research object of PISA are students from 15 to 16 who are about to complete
full compulsory education. The purpose of scientific literacy assessment is to
identify the key factors of different levels of teaching, curriculum and
learning environment, to explain the differences in students' scientific
performance, so as to judge whether these students have good scientific
literacy to deal with the challenges posed by modern social situations (for
example, health and energy) and then provide solutions to the global science
and education reform (OECD, 2016).
It can be seen that the evaluation of PISA is based on context, rather than
just the degree to which students master the subject knowledge in school class.
PISA pays more attention to apply the knowledge and skills students have
mastered to real life and focus on the examination of students' ability.
4.3
The Evaluation Concept and Purpose of TIMSS2023
The theory of TIMSS evaluation is the
"curriculum hierarchy theory" of Goodlad,
which is improved to propose Intended Curriculum, implemented curriculum and
Attained Curriculum to apply the three-level curriculum model to scientific
evaluation (Pan et al., 2018). It can be seen that TIMSS evaluation starts from
the course, and believes that the course is the main reason for affecting
students' academic performance. The purpose of the evaluation is to investigate
the students' mastery of the learning course knowledge, test the gap between
the acquired course and the target course, and analyze the problems, so as to
provide reference for the national education decision-making.
4.4
Comparison of Evaluation Concept and Purpose among NAEP2019, TIMSS2023, and
PISA2025
In general, the evaluation concepts of the
three major evaluation projects are different, but the overall goals are the
same. On the one hand, it is to evaluate students' achievements after studying
in school, so as to understand the educational achievements of each country or
district. At the same time, it explores the factors that affect the
effectiveness of education to provide guidance for improving education. On the
other hand, they all reflect the different requirements of social and national
curriculum for the development of students 'scientific literacy, and transform
the requirements into specific scientific literacy and evaluation content
framework system, so as to realize the effective monitoring of students'
scientific literacy and science education and teaching quality. In addition,
there are also certain differences in the evaluation purpose. PISA pays more
attention to the role of science education in the society, so in the assessment
will determine the relevant scientific literacy influencing factors of
information sources in order to clarify the different factors in the education
system. But TIMSS and NAEP are more inclined to the specific implementation of
curriculum requirements.
5.
EVALUATION FRAMEWORK AND CONTENT
5.1
Evaluation Framework and Content of NAEP2019
The scientific evaluation of NAEP2019 is
divided into two dimensions: scientific content and scientific practice. The
scientific content consists of a series of key facts, concepts, principles,
laws, and theories describing the natural science, the life science, and the
earth and space science. Scientific practice consists of four sub-dimensions:
identifying scientific principles, applying scientific principles, using
scientific investigation and using technology to design. NAEP science content
examines what students know about scientific knowledge. Scientific practice
examines how students can use scientific knowledge to solve practical problems.
The above two points are consistent with the NAEP assessment philosophy.
5.2 Evaluation
Framework and Content of PISA2025
PISA2025 Scientific evaluation is divided
into three dimensions: scientific knowledge, scientific ability, and scientific
identity. Science situation is no longer a dimension of PISA 2025 assessment,
but implicit in the dimension of scientific knowledge, scientific ability and
scientific identity.
In terms of
scientific knowledge dimension, the content knowledge, procedural knowledge and
cognitive knowledge proposed by PISA2015, adding three dimensions of social
environment system and sustainability, the development and abuse of scientific
knowledge, and informatics. As for social environment system and
sustainability, which is a comprehensive knowledge field designed to support
students to solve complex problems closely related to social life. The
development and abuse of scientific knowledge expects to make students better
understand how knowledge develops and help them avoid knowledge misuse.
Informatics studies the data, structure, and behavior in natural and computational
systems. Students with digital information literacy will be able to understand
AI concepts and computing systems at a basic level in order to act and make
decisions based on the information provided to them.
In terms of
scientific ability dimension, which retains scientific interpretation
phenomenon from PISA2015 and expands the two existing abilities of evaluating
and designing scientific inquiry, scientific interpretation of data and
evidence, and adds two new abilities of using scientific knowledge for
decision-making and action and using probabilistic thinking. Using scientific
knowledge for decisions and actions aims to evaluate students' ability to use
scientific knowledge to solve problems creatively. Students need to think about
the different mechanisms that lead to specific events, and the extent to which
these events are caused by the scientific, social, and economic aspects.
Probabilistic thinking mainly refers to the possibility of estim
specific results using mathematical logic tools.
Scientific
identity is a new dimension from PISA2025 scientific literacy assessment
framework, which means that students feel a meaningful connection with science,
a certain interest in science, and a sense of identity to participate in
science in daily life. The evaluation content of scientific identity in the
PISA 2025 scientific literacy assessment framework includes four aspects:
scientific capital, critical scientific users, inclusive scientific experience
and practice, moral ethics and values. Scientific capital refers to the
comprehensive measurement of students' scientific identity, scientific
attitude, behavior and connection, and finally forms the sense of science
"serving themselves". The ability of critical science refers to the
critical use of science and other expertise to contribute to individual and
social interests, especially in addressing social inequalities. Scientific
experience and practice involves addressing and narrowing "gaps" in
scientific knowledge, abilities and identity among different social groups,
allowing students from diverse backgrounds to experience an inclusive science
learning environment. Ethics and values refer to the ethics and values based on
their identity and understanding of scientific knowledge.
To sum up,
PISA2025 evaluation framework emphasizes the interdisciplinary integration of
scientific knowledge, the scientific ability of social decision-making needs,
and the scientific identity that lays the foundation for learning achievements.
5.3
Evaluation Framework and Content of TIMSS2023
The scientific evaluation of TIMSS2023 is
divided into three dimensions: scientific content, scientific cognition and
scientific practice. But the evaluation content is mainly organized into
scientific content and scientific cognition and supplemented by scientific
practice. Therefore, the evaluation of scientific practice is not carried out
separately, but by some evaluation questions in the content and cognitive
field.
The fourth grade science content field of
TIMSS2023 includes three topics: life science, material science and earth
science. The eighth grade science content of TIMSS2023 covers four topics:
biology, chemistry, physical science and earth science. The difference in
content areas between the fourth and eighth grades reflects the nature and
difficulty of the science taught in each grade. And each theme is further
described by specific objectives that represented the expected knowledge,
abilities, and skills that the students assessed in each topic.
The percentage of goals in the scientific
cognitive field varies in the fourth and eighth grades, but involves the same
topics, namely knowledge, application and reasoning. The ability of knowledge
requires students to recall, identify, describe, and provide examples of facts,
concepts, and procedures. Applications focus on relating scientific knowledge
to a specific context, as well as on solving practical problems. Reasoning
involves the use of arguments and scientific understanding to analyze and
generalize, ordinarily in unfamiliar situations and complex environments. All
of these topics include projects developed for the three cognitive domains. For
example, the life sciences field includes knowledge, application, and reasoning
projects, as well as in other content areas.
The field of
science practice points out that participating in scientific practice allows
students to understand how scientific activities are carried out, and then to
understand and appreciate the nature of science and scientific knowledge.
Scientific practice is the foundation of all scientific disciplines and
includes skills in daily life and learning. The process of scientific practice
is usually nonlinear and proceeds in an iterative manner. Including five
activities, first, ask questions based on observation and theory; second,
design the survey and generate data; third, use data; fourth, answer research
questions; and fifth, demonstrate based on arguments.
All in all,
the scientific content of TIMSS2023 scientific evaluation points to the theme
of evaluation. Scientific cognition focuses on the thinking process and
scientific practice emphasizes the use of scientific skills to carry out
scientific inquiry.
5.4
Comparison of Evaluation Framework and Content among NAEP2019, TIMSS2023, and
PISA2025
The three major evaluation projects are
mainly based on scientific knowledge and scientific ability. In the dimension
of scientific knowledge, it shows the consistency of covering the three basic
fields of matter, life, earth and space. However, PISA divides knowledge into
content knowledge, procedural knowledge, cognitive knowledge, social
environment system and sustainability knowledge, the development of scientific
knowledge and its abuse and informatics knowledge. TIMSS, which emphasizes
courses, lists individual courses, such as chemistry, while NAEP and PISA
incorporate chemical knowledge into the physical realm. In the dimension of
scientific ability, the three major evaluation projects all focus on scientific
inquiry and scientific practice. However, PISA and NAEP focus on the embodiment
of students' scientific ability in the process of scientific practice, and
organically unify the thinking process and practice process. As for TIMSS, the
evaluation of scientific practice is not carried out separately, but
implemented with some evaluation questions in the content and cognitive field,
which shows the separation of thinking process and practice process to a
certain extent. This can also be explained that TIMSS puts more emphasis on
thinking process.
6.
THE ENLIGHTENMENT OF THE INTERNATIONAL SCIENCE LITERACY ASSESSMENT PROJECT
6.1
Clarify the evaluation purpose and construct the scientific ability framework
The three major evaluation projects have
different emphasis on scientific literacy evaluation, but the overall goal is
the same: to evaluate, analyze and compare students' educational achievements,
so as to provide data support for the release of national education policies.
Currently, science education no longer emphasizes students 'memory of scientific
knowledge, but turns to cultivating students' scientific ability. However, in
some countries, the trend of shifting from students 'scientific academic
achievement evaluation to students' ability examination is more obvious, but it
has not yet formed as mature framework as PISA. Therefore, on the basis of
clarifying the purpose of science education, the relevant educational
researchers should pay attention to refining the connotation of scientific
ability, which will become the basis for guiding the front-line teachers to
carry out science education.
6.2
Innovate the types of test questions and strengthen the application of real
situation
Teachers' personal evaluation and paper-pen
test are mainly used for scientific evaluation. Most scientific situations are
applied to science classroom teaching, such as video import is used to
introduce relevant content, but the teaching of new knowledge is gradually
separated from the real situation. The three major evaluation projects all
attach great importance to the application of real situations and PISA
specially divides scientific situations into three categories: individual,
local / national and global applications. The ultimate goal of learning is to
use knowledge. In the classroom, teachers can make use of the digital
environment and adopt interactive teaching methods to improve students'
initiative and enthusiasm, and further guide students to solve practical
problems with the knowledge they have learned.
6.3
Attach importance to the law of development and observe the level of cognitive
development
The respondent of PISA2025 included students
aged 15-16 who are about to complete full compulsory education, TIMSS2023
surveyed students in fourth and eighth grades, and NAEP survey included
students in grades 4,8, and 12 in the United States. The brain development of
the fourth grade students is just in the critical period of internal structure
and function perfection, and their physiological and psychological
characteristics change significantly, which is the best period to cultivate
learning ability, emotional ability, will ability and learning habits. Junior
high school students are in the period of "physical and mental
fusion", among which the eighth grade students are particularly obvious,
and the education circle calls it the "psychological weaning period".
From the selection of the survey objects of the three major evaluations, we can
draw a conclusion that whether in the selection of science teaching content and
methods, or in the cultivation of scientific ability and thinking, the law of
students' physical and mental development and the principle of teaching
students should be in accord with their aptitude. And targeted teaching should
be carried out according to the different characteristics of students.
6.4
Collect background data and provide comprehensive educational information
The Background survey of PISA includes:
school questionnaire, student questionnaire, educational career questionnaire,
information and communication technology proficiency questionnaire, parent
questionnaire, science teacher questionnaire, general teacher (non-science
course teacher) questionnaire (Shi &
Lin, 2020). TIMSS collects important information related to policy through
investigating questionnaires, which filled out by students and their parents,
teachers and principals about students' experiences in school and home. And
then study their impact on student performance (Zeng & He,2020). The background survey of NAEP includes three
parts: student situation, school situation and teacher situation (Liao et al, 2016). Studying the link
between background data and student academic achievement not only help the
respective states continue to conduct the monitoring of student performance to
improve teaching but also indirectly monitor the development of education
quality in their own states.
6.5
Pay attention to the trend of The Times and adopt various evaluation methods
In the information age of knowledge
explosion, it has become the trend to replace the traditional pen-and-paper
test with the computer evaluation questions for large-scale evaluation items.
PISA, TIMSS and NAEP already has transitioned from paper-and-pencil testing to
computer evaluation. Compared with traditional paper-and-pencil tests, computer
evaluation questions have the following advantages: first, digital evaluation can improve the way
of data collection and give a mark automatically. Second, it can reduce the
printing and delivery costs of paper delivery, thus improving the efficiency of
evaluation. Third, computer evaluation can develop interactive test questions.
In PISA evaluation, students can conduct simulation experiments in a computer,
which is closer to the real purpose of PISA examination. Based on the above
three reasons, it is of certain positive significance to pay attention to the
computer development and evaluation questions and adopt various evaluation
methods to evaluate students' scientific literacy.
Competing interests
The authors declare no competing interests.
Acknowledgements
We would like
to thank the financial support from the Zhejiang Social Science Federation
(2023N079) and the Graduate Scientific Research Foundation of Wenzhou
University (3162023004018).
REFERENCES
Creemers, B. & Kyriakides, L. (2008). A theoretical
based approach to educational improvement. Establishing links between
educational effectiveness research and school improvement. Yearbook on School
Improvement, 2, 1-25.
Deng, X. Z., & Yu, R. W. (2022). The Analysis and Enlightenment of
NAEP Scientific Assessment. Biology Teaching in
Middle School, 28, 65-68 (in Chinese).
Gu, N. J. (2023). The overview and
enlightenment of the scientific evaluation framework on TIMSS2023 and compared
with the TIMSS 1995-2019. Examination Research, 01,
75-84 (in Chinese).
IEA. (2023).
TIMSS 2023 Assessment Frameworks. https://timss.bc.edu/timss2023/frameworks/index.html
Liao, B. Q. et al.
(2016). The analysis and implications of questionnaire design for international
education quality monitoring. Foreign Primary and
Secondary School Education, 05, 15-21 (in
Chinese).
Liu, K. W., & Liu, S. (2011). The content and
characteristics of PISA2015 on the scientific literacy assessment. Comparative Education
Research, 07, 98-106 (in
Chinese).
Lu, Y. T. (2007). The comparative studies of students' academic achievement in the
context of globalization on TIMSS and PISA. Examination research,
03,76-92 (in Chinese).
National Assessment Governing Board. (2019). National Assessment
Governing Board. Science Framework for the 2019
National Assessment of Educational Progress.
Washington: Government Printing Office (U.S.).
OECD. (2016).
PISA 2015 Assessment and Analytical Framework：Science,
Reading, Mathematic and Financial Literacy.
Paris: OECD Publishing.
Pan, X. M. et al.
(2018). A study on the scientific literacy of compulsory education students and
its key influencing factors based on international assessment experience from
PISA, TIMSS and NAEP. Foreign Education Research, 45,
76-86 (in Chinese).
Shi, X. Y., & Lin, J. (2020). PISA questionnaire design based on
PISA2021 questionnaire framework. Shanghai
Education and Research, 07,
55-59 (in Chinese).
The State Council of China. (2021). The Outline of the Action Plan for
National Scientific Literacy (2021-2035). Bulletin of The
State Council of the People's Republic of China, 19, 12-20 (in
Chinese).
Zeng, X. P., & He, T. (2020). The
new trend of international Mathematics and Science Education evaluation based
on example Analysis of the main characteristics of TIMSS 2023. Basic Education Course, 17, 67-71 (in Chinese).
Zhang, X. M. et al. (2011). The professional vision of PISA. Education Research, 06,
3-10 (in Chinese).
Zhou, H. (2005). Generation and development system of the National
Assessment of Educational Progress (NAEP). Foreign Education Research, 02, 77-80 (in
Chinese).
