Measurement & Evaluation
Why Measure and Evaluate Your Group?
Local groups are a really important part of the EA movement, through directing funds, spreading EA ideas and changing people's careers. But, we can make an even bigger impact if we understand which strategies work best.
Self-assessments help to improve your understanding of the members within your group and what they want out of it, improve awareness of possible issues to allow for mitigation, and help improve your group's strategy in planning events.
Having information on what you have done that has worked or not worked helps the movement learn from best practices, advice and tips.
Many useful innovations have come from local groups, and it is better for the movement if we can share this information clearly with each other, and not be constantly reinventing the wheel.
Improve budgeting and making sure that we are spending our time and resources doing things that we know are more likely to be effective.
Possible Goals
The goals of measurement and evaluation will be very dependent on your group, on how big it is, what types of events you hold, what purposes you hope to achieve etc. How much effort you put into evaluation, and exactly what to measure, is something that you should discuss with your core members, so that you can make sure what you get out is going to be valuable.
From 2020-2022, CEA was focused on creating more highly-engaged EAs, and we generally recommended groups to consider making this a central goal of theirs as well. We've since moved away from focusing heavily on this metric, but groups can still consider using or modifying it.
Highly-engaged EAs are people who create significant behavioural changes based on EA principles, such as shifting their career path and/or taking jobs or internships directly relevant to EA.
As we mention on our Strategy page, it's important for groups to focus on their comparative advantage, so it's possible highly-engaged EAs may not be the best metric for you to track. Other possible goals for your group could be:
Helping people hear about EA. Getting people in your local community to learn about EA for the first time and have a reasonably accurate idea of what it means.
Getting people to engage with EA for the first time. Having community members attend an EA event, or engage with EA in some preliminary way, and come away with a positive impression and motivation to come back.
Connecting people with the community. Having community members understand the core ideas of EA well, be motivated to act on those ideas, or make a commitment to acting on those ideas.
Below we suggest metrics for these goals, how to collect data in relation to them, and how to interpret the data.
Possible Metrics
No single metric is perfect, so it is better to collect a range of them, use the metrics as best as we can, and adapt what and how we collect information in response to feedback and changing circumstances.
Below are relevant questions for each of the four aforementioned stages of community-building, along with some of the most broadly-useful metrics to answer them.
How many people have spent x hours engaging with your group in the last 12 months?
How many people have spent at least 10 hours engaging with effective altruism content? How many people have spent at least 100 hours engaging with effective altruism content?
How many people have used the principles of effective altruism to...
Choose where to donate?
Develop their career plans?
Volunteer on an EA-related project?
Apply for specific internships or jobs?
Choose their degree programs?
How many people in your group have taken the Giving What We Can pledge?
Suggestions for Collecting Data
Feedback Forms & Surveys
Use feedback forms and surveys if you want self-reported assessments from your community members or event attendees.
Use wording that avoids ambiguity and too much slang (perhaps by asking a non-EA friend to give you feedback on the language used).
Think carefully about how frequently you use forms and surveys. Having feedback forms at every event may be completely reasonable (though may appear unusual, given many non-EA organisations don’t do this); however, sending out surveys separately from events too often may lead to survey-fatigue, and reduce the response rate.
Make sure the forms/surveys aren’t too long. If using online, avoid making too many questions required, so that respondents are more likely to skip questions than abandon the form.
Here's an example of a survey to use after an event:
Examples of Write-Ups
These are write-ups from different local groups, evaluating the success of a range of activities. Writing up your quantitative and qualitative assessment of your activities may help you think through your future strategy plans. Sharing these write-ups with other groups can also be helpful to avoid the duplication of labour and to save repeat mistakes from being made.
Annual Reviews
EA London Impact Assessment 2017
EA Berkeley Reports (2015, 2016, 2017)
Giving What We Can + 80,000 Hours Cambridge Yearly Review ('16/'17)
80,000 Hours Cambridge Yearly Review ('15/'16)
Casual Reflections
Reflections on Running EA Seattle