How to use monitoring and evaluation to reflect on your mission

We needed to talk about our mission to set discussions of monitoring and evaluation in a context. An organisation’s mission explains the reason for its existence. It outlines the purpose and direction of the for staff, volunteers, participants, other professionals and funders. SLT’s mission is; to ensure that local people, especially those experiencing social isolation or mental distress, have access to the support they need for better health and wellbeing.

How do we define Social Isolation?

Social isolation can be defined as ‘The inadequate quality and quantity of social relations with other people at the different levels where human interaction takes place (individual, group, community and the larger social environment).’
Social isolation differs from loneliness. Loneliness can be defined as ‘An emotional perception that can be experienced by individuals regardless of the breadth of their social networks.’ 1.

People at risk of Social Isolation

SLT is an Islington based organisation , with a pan-London reach. The groups most at risk of social isolation and poor mental health according to the state of equalities report published by Islington in 2020 are:

• Trans people are more likely to experience poverty, discrimination, and mental health problems. The 2018 Stonewall study found that more than 28% of trans respondents who were in a relationship in the last year had been subject to domestic abuse and 25% had experienced homelessness at some point in their lives.
• Research has found that there are two other major mental health issues which disproportionately affect LGB people: alcohol issues in lesbian and bisexual women and body image issues for gay and bisexual men.
• Individuals who interact with the criminal justice system are particularly vulnerable to mental ill health. Up to 90% of prisoners have some form of mental health problem, and 10% of male and 30% of female prisoners have previously experienced a psychiatric acute admission to hospital.
• Nationally, 13% of adults with a disability reported feeling lonely often or always in 2017/18.

• Refugees in the UK suffer disproportionally with poor mental health. The majority (61%) of asylum seeks in the UK experience serious mental distress.
• Older people over the age of 65

How do we define mental distress?

According to the Counselling Directory, “the term ‘mental distress” is used to describe a range of mental health issues, from the more common problems such as anxiety and depression, to the less common, such as schizophrenia.”
Common symptoms of depression include low mood, fatigue and loss of interest or excitement in things previously enjoyed. Often, symptoms of depression will impair emotional and physical well-being, as well as the behaviour of the person.

A 2016 Time to Change survey of over 7,000 people living with mental health issues found that 64% were feeling isolated.

The Changing Needs of M&E for SLT

In February 2018, SLT’s Board agreed an ambitious 3 year plan. The main focus being:

1. To maintain the frequency and high quality of SLT’s present activities
2. To bring in activities for younger people via feasibility studies so that services can be provided that will attract this population.
3. To increase Saturday Project workshops to 52 weeks a year. SLT has already made some headway here with 28 Saturdays provided last year.
4. To develop, in partnership, a volunteer befriending service, so that participants can be supported outside of activity times.
5. Once the befriending service is set up, to endeavour to link in more closely to GP’s and health professionals so that more vulnerable residents can be reached.

Other than the above aims in the 3 year plan, SLT has a wider ambition to continuously provide the best possible service to participants in accordance with it’s outcomes framework. All SLT’s outcomes fall under 3 categories. Each category requires a different monitoring & evaluation approach so it is useful to separate these out into:

Impact (service effectiveness) – a service’s impact is defined by the change, effect or benefit that results from the services or activities on a wider society than its direct users. It is often long term, broad and sustainable.

Engagement (service satisfaction) – a service’s level of engagement is defined by the effectiveness of meeting participants expectation / need. This can be measured by enjoyment, satisfaction, attendance, returning visits etc.

Delivery (service outputs) – a service’s delivery outputs the activities, services and products provided by the organisation. This can be measured in volume / frequency etc.
Data that SLT collects will be based on key indicators that help to measure these outcomes.

SLT’s specific aims with respect to the above outcome categories are outlined below.

Why are we reviewing the existing M&E?

Historically SLT have monitored its services through:

Participant feedback forms – SLT participants have been asked to complete paper feedback forms after every Friday Evening Event, Saturday Workshop, and all other activities. This helps us to understand participants enjoyment of the event, the food (if relevant), the physical space, as well as obtaining qualitative feedback on what was good, and what could be improved.

Volunteer feedback forms – the core kitchen and assistant volunteers at the Friday Evening Events are asked to complete a paper feedback form after every Friday Evening Event. This helps us to understand how the volunteers were, any issues that arose, or extra support or needs from the core team.

Session Leader feedback forms – these are completed after each session by the session leader. It includes a summary of the event, and magic moments observed.

Registration forms – each new participant is asked to complete an equal opportunities registration form. This allows us to understand if our participants identify with lived experience, ethnicity, and live in certain boroughs. We also ask if there is anything we need to know to best accommodate them in our service (e.g. if ramps are required). All questions are optional but help us to understand our reach, and ensure we know how participants would like to keep informed.

SLT decided to review their M&E processes in 2019 as it felt that we were capturing a lot of data and information from our participants, but weren’t able to use a lot of it in a more meaningful way to understand our impact.

For example one question on the feedback forms asks about ‘Enjoyment Factor’, this feedback was very consistently positive, thus not helping us learn much about the nuances of our service’s impact except that participants enjoyed the events. We wanted to review our questions to ensure that the data we gather is more informative, continues to place participants at the heart of our work and demonstrates our impact.

What are the challenges of monitoring and evaluation?

We received feedback from those who completed SLT feedback forms regularly that revealed that completing the paper forms after each activity can feel repetitive. Our participants also felt that they rarely saw the effect of their feedback; was it being digested by SLT? And what was SLT doing with the feedback? We want to make sure that as an organisation we only ask for information that we will use to inform our service, our funder needs, or other decision making.

Another challenge is about how we monitored our reach. We had previously only been asking for very basic information from our participants which did not easily allow us to identify if we were reaching people experiencing mental distress & isolation or loneliness. For example we invited participants to let us know if they had a disability that would be worthwhile for us to know, but did not specify about mental health.

The final challenge, was that by using 4 different types of feedback forms from participants at various stages, we did not have consistent data that was easy for us to analyse. The questions on each feedback form were slightly different from each-other. In addition, we were collecting a lot of feedback from Friday Evening events but not from other activities.

Read our next blog to find out how we tried to overcome these.

How you can use monitoring and evaluation to reflect on your mission

Invite key stakeholders to a discussion about your mission. These could be your staff, volunteer reps, client reps and trustees. Ensure that there is a diverse set of voices in the room that represent difference experiences across the organisation.

• Define the key concepts with you r mission
• Assess whether your current processes are generating data that is useful and necessary
• Identify your challenges and think about how you can overcome these

References:

1. Zavaleta D, Samuel K, Mills C. Social isolation: a conceptual and measurement proposal. POPHI Working Paper No 67, Oxford Poverty and Human Development Initiative, 2014. (as referenced LINK )
2. https://www.islington.gov.uk/~/media/sharepoint-lists/public-records/communications/information/adviceandinformation/20192020/20200131stateofequalitiesreport20201.pdf
3. Bachmann, CL and Gooch, B (2018). LGBT in Britain: Trans report. Stonewall and YouGov. (as referenced HERE)
4. RARE Research Report, PACE (2015) ( & referenced HERE)
5. Singleton et al., 1998; Department of Health (2007) (as referenced HERE)
6. ONS. Annual Population Survey: Disability and Loneliness (2019). (as referenced HERE)
7. UK Refugee Council, 2019 (as referenced HERE)
8. https://www.counselling-directory.org.uk/mental-distress-stats.html
9. https://www.counselling-directory.org.uk/mental-distress-stats.html
10. http://www.time-to-change.org.uk/news/englands-biggest-ever-survey-state-stigma