T.A.A.F.E. Towards an alpine Age-Friendly Environment
Dear reader, welcome to the fifth TAAFE newsletter, where we will give you some information about the evaluation model

The TAAFE Evaluation Model

 

The TAAFE evaluation activities are being conducted through methodologies that integrate qualitative and quantitative tools— with a multidimensional approach to age-friendly environments. Based on the general approach of the TAAFE project, the impact was assessed thanks to indicators selected starting from the 8 dimensions identified by the World Health Organisation in the framework of the project "The Global Age-Friendly Cities” on policies, services and facilities promoting active-ageing processes.

The TAAFE evaluation model consists of three instruments:

  1. “pre-post" analysis to measure the impact of the participatory methodology on the level of participation in each pilot site;
  2. Social impact evaluation through a SWOT analysis to recognize strengths & weaknesses and opportunities & threats of the pilot actions and to highlight the changes created by the project;
  3. SROI to test of the social impact assessment model of the TAAFE model.

 

 

In this regard, to assess the short- and medium-term effects of the TAAFE activities, the “prepost" evaluation was developed with the target objective of measuring the impact of the participatory methodology on the level of participation of the citizens, professionals and members of organisations involved by each partner in  their pilot action.

 

Through data collected with ad-hoc questionnaires, for each indicator, and for each dimension in aggregate terms, the measurement of impact was based on a Cantril scale that allowed respondents to evaluate, with a score from 1 to 10, their level of participation in participatory planning and/or decision-making process before and after the implementation of TRIOs at local level.

The designed questionnaires were administered to participants in local TRIOs and involved stakeholders at the beginning (baseline questionnaire) and at the end (final questionnaire) of the planned TRIO activities.

 

The baseline survey involved 96 people in the whole project area. In the baseline questionnaire 75 respondents out of 96 declared that they had already started participating in the project’s activities— with 53 of them being senior citizens (aged 60 years and older) and the majority of respondents had been involved as Local action group members. The analysis of the aggregate data of all pilot sites indicated substantial congruence between past experiences and expected impact within the project for all dimensions; “Communication and Information”, “Community support and health services” and “Social Participation” were the  dimensions that received the highest average value.

The final questionnaire was focused on the dimensions, and related key aspects, chosen by the partners for the implementation of their pilot action. The impact of the participatory methodology that each partner implemented in the pilot actions is still underway - as of beginning of June 2022 - and is being measured by calculating the gap between the scores assigned in the baseline questionnaire and in the final questionnaire for the dimensions related to the pilot actions for each individual partner.

Considering that evaluation activities are still underway, the initial analysis of the data collected allowed for some methodological considerations that has to be taken into account when elaborating the final data collection.

First, the COVID-19 health emergency strongly affected the participatory methodology— with the imposed restrictions considerably reducing the possibility of having face-to-face meetings between TRIO members, Local Action Groups and stakeholders involved in the activities, thus considerably weakening the positive effects of participation itself.

 

The second relates to the large turnover that each country had among the participants from the stakeholders who participated in the pilot action activities— preventing in some cases continuity in the development of the different lines of action proposed by project partners. For this reason, an assessment of the results of the qualitative survey should complement the analysis of the quantitative results— resulting in additional information to properly evaluate the benefits of the TAAFE participatory methodology.

Complementary to the pre-post analysis, the social impact evaluation aimed to collect  qualitative information on  the pilot actions through  a SWOT analysis focused on the activities and solutions adopted in the different pilot sites.

 

Data collection was carried out through an open-ended questionnaire asking TRIO members to identify and clarify their opinions regarding strengths, weaknesses, treats and opportunities of the pilot action— both in relation to the activities carried out and to specific conditions impacting participatory processes.

 

Some initial results are summarized in the table below:

In addition, the social impact evaluation collected information about the activities carried out by the respondents in TAAFE—asking them to link these activities with changes for their organisation and senior citizens.

Some examples of relevant activities indicated by respondents include the training of volunteers for quality ageing, courses for informal caregivers, the creation of a weekly meeting called “Digital Café” and the organization of digital support activities. Among the related changes, respondents indicated more intergenerational cooperation, increased access to literature, better knowledge of institutional functioning and better cooperation with representatives of the municipality.

 

To complete the SWOT analysis, a final open-ended question concerned other activities outside of the TAAFE project in which respondents participated thanks to their experience with the pilot action implementation.

 

Furthermore, the analysis of the answers from partners is been used to develop a map of the impact dimensions implemented in the different pilot actions. This map will allow the identification of some indicators for the quantification of the impact dimensions. An evaluation model based on the identified indicators for the assessment of Age-Friendly Environments will be developed.

 

Finally, a test of the social impact assessment model was carried out using the Social Return On Investment (SROI) technique. To clarify, the SROI allows measuring the correlation among benefits and costs incurred for a given initiative— with the measure being expressed by a proportion among the invested amount and its social return.

 

The SROI methodology highlighted the adoption of the stakeholders’ perspective— providing a comprehensive evaluation by measuring the outputs based on the objectives of the stakeholders that participated in the TAAFE activities.

Given the timeframe of the project activities, the SROI test was carried out only for the Italian pilot site.

The application of the SROI analysis was structured into 6 phases:

 

  1. Description of the field of analysis: analysis of documentation and context of the pilot action
  2. Identification of the main stakeholders of the pilot Action: analysis of their main characteristics and criteria for their inclusion/exclusion
  3. Development of a theory of change for stakeholders: the changes that can be achieved by the pilot action are described for each stakeholder. The dimensions of impact are defined.
  4. Definition and measurement of outcomes: the indicators are selected and financial proxies are introduced
  5. SROI calculation: a SROI ratio is calculated on the basis of the quantification of the impact indicators
  6. SROI report: a reporting document is prepared

 

The partner responsible of the TAAFE evaluation activities is the Central European Initiative  - Executive Secretariat.

For any further information contact:

Gian Matteo Apuzzo

Senior Programme Manager, Central European Initiative

apuzzo@cei.int

TAY TUNED!

 

Stay tuned to keep updated on the progress of the activities!

 

You can visit our webpage or follow us on   LinkedIn and   Twitter

More
   
European Regional Development Fund