Beyond good grades: Evaluating kids’ compassion in Catalonia

The debate about the need to develop today’s schools towards a more holistic model of education is not new. However, few institutions have taken action to scientifically analyse whether current methodology is working or not, beyond the evaluation of academic results.

Even fewer institutions have obtained real data that shows whether their educational model is achieving the transformation that they defend in their mission or educational policy. 

At the moment, the reality is that they do not know whether they are forming better people capable of changing the world or not.

Are they socially engaged, compassionate, competent young people who are conscious of the role they play in society?

How can you obtain data that certifies the degree of compassion of an 18-year-old or shows whether this has been gained thanks to any educational model in particular? How can you test a methodology to see if it has achieved a real impact on the development of qualities such as generosity or leadership? These are intangible values whose impact is extremely difficult to measure. It is also a risk for those projects prone to the possibility of achieving unfavourable impact results.

The “Let’s Improve” project

Nevertheless, it would seem that some light is being shed on some of these questions. The Fundació Jesuïtes Educació – a Catalan Jesuit education foundation – has been working on a new integrated model of education called Horizon2020 in its schools since 2014. 

At the same time, it is developing a parallel project in the field of educational impact measurement on a regional level. It’s called “Millorem”, or “Let’s Improve”, and scientifically analyses what kind of people the 800-plus young people who leave the eight Jesuit schools in Catalonia after their secondary education are. Are they socially engaged,compassionate, competent young people who are conscious of their own reality and the role they play in society?

To back up this change in their management style and their educational model, the foundation has developed a tool in conjunction with social consultancy Stone Soup Consulting which measures how the acquisition of these values influences the behaviour of the students. These values are referred to as the Four Cs: consciousness, compassion, competence and commitment.

The knowledge acquired on completing the first phase of this project is already being transferred to other European projects which can have an impact on the development of people’s competences and which also want to measure possible changes in their behaviour.

As Susan Balet, one of the Stone Soup coordinators, affirms, the “Let’s Improve” project has “opened a window in the analysis of the impact of education which allows us to be optimistic and think that if we continue in this direction we will be able to know the real influence of education on the behaviour and personal development of students.”

Seven months’ work; 100 professionals

The first step of this educational transformation began to take shape more than five years ago when the foundation began working on a model for measurement which allows the gathering of information beyond the resources, activities or results obtained.

According to Luis Ylla, the director of planning and improvements at the foundation, “the analysis of the impact of our educational model was not systematic and the indicators were weak. We wanted to measure the tendencies in the young people’s behaviour in relation to the 4Cs and identify areas for improvement.” The foundation needed to know what was working and what wasn’t to be able to improve their results.

For Ylla, “the most complicated part was the elaboration of a reference table which could associate the 4Cs with certain behaviour and was based on scientific and technical grounds which would endow it with the necessary consistency.”

This measurement of the young people’s competencies is carried out through a 15-minute survey, complemented with information extracted through focus groups with teachers and students. The development of the survey took seven months’ work and the coordination of more than 100 professionals, including the project coordinating team, tutors, statisticians, members of the governing board and foundation, psychologists, students and education professionals.

What’s more, once it was approved by the experts, the survey had to adapt complex technical concepts to the language and habits of 18-year-olds. According to Balet, this condition was indispensable to achieve the greatest number of genuine answers.

The stakeholders have a lot to say

“Let’s Improve” counted upon several students who helped to develop the survey. After examining the draft copy of the first survey the students found it too long, occasionally incomprehensible and it even raised their suspicions. 

Was it necessary to ask them about their consumption of drugs or their sex lives to find out their degree of conscience or self-care? 

Thanks to their participation, various questions became the subject of debate, were re-phrased or were eliminated and the language transformed. In the same way, the students contributed ideas about the most efficient channels of communication and how to use new technology to achieve the greatest number of replies. 595 students answered the survey through the school intranet and their mobile phones and the project was explained in an online video.

“Let’s Improve” also had to fit into a school ecosystem with multiple interest groups and diverse educational needs. The foundation wanted to involve all the secondary level tutors and directors in the eight Jesuit schools in Barcelona. As far as those responsible for the project were concerned, it was fundamental that all the people involved took the project on board to be able to manage the transformation from the inside. 

The results obtained from the surveys completed by the first generation of second-year baccalaureate students showed a degree of acquisition of the 4Cs, in most cases, higher than 60 points out of 100. This score increases even after they leave school, which was shown by a second survey of the same students a year after they finished their school studies. Only by analysing the behavioural tendencies of different generations will the foundation know whether the educational changes being introduced in the model have a positive influence and increase the degree to which they make its system of values their own.

It is, without doubt, a risky gamble which could have shown undesirable results but, according to Ylla, “this possibility didn’t worry us and we had low expectations. We needed to know if what we’re doing is useful and thus do it better and pass from intuition to reality.” 

For her part, Balet says: “From Stone Soup we always maintained that this was an experiment meant to measure the process of change and create a living tool which could be adjusted over time and continue to measure the behavioural tendencies of future generations.”

What’s all this for?

This ground-breaking project is already influencing the management of various projects in the social and educational fields. Balet says: “'Let’s Improve’ has contributed to important changes by measuring intangibles which can act as a lever for change in the educational sector as a whole.” Stone Soup is already transferring the knowledge gained to other projects with similar objectives in matters of socially responsible behaviour.

Meanwhile, for the foundation, now is time to analyse the results of the surveys from the different generations of students and gradually impregnate the new educational model with the conclusions. What’s more, we can now envision the creation of a national team capable of incorporating the methodology into other regions and improving this tool. This is undoubtedly a hopeful step in the direction of the development of the educational sector on a national scale.

What remains to be done is probably the most complicated part: make concrete decisions depending on the results of the impact achieved and improve the current educational model. This is the only way we can move towards a system capable of forming people who are not only academically well-prepared, but are also more socially responsible.