I recently spent an informative and enjoyable afternoon at the CLINKS event “Justice Data Lab – one year on”.
The Justice Data Lab is an exciting step forward in making use of government data to better understand what works in reducing reoffending. It is part of a wider, ambitious project led by NPC to open up government data to the not-for-profit sector to help them understand the impact of their work.
A key, but challenging, element of the Fulfilling Lives: Supporting people with multiple needs evaluation is using administrative data, on offending, benefits, use of health services and so on to estimate the cost of supporting people with multiple needs, and evidence the impact of the programme. So we will be keeping a keen eye on how this work develops over the coming months and years and considering ways the evaluation might benefit from it.
The first half of the event was dedicated to providing information on the data lab, how it works and how projects can use it. Nicola Webb of the Justice Data Lab explained all. Projects working to reduce offending provide some key information about their beneficiaries. A minimum sample of 60 is needed as a smaller sample is unlikely to produce statistically significant results. The Justice Data Lab team identifies the beneficiaries within their data, and gathers together data for a similar group of people who have not been helped by the project to create a comparison group. They then analyse the one-year proven reoffending rate for both groups to determine whether there is a significant difference between those receiving the intervention and the comparison group. In the interests of openness, all reports are published by the Ministry of Justice, irrespective of whether the results are positive, negative or inconclusive. A weakness however is the limited data which can be used to create a comparison group – as a result analysis for beneficiaries with a history of mental ill health or substance misuse may not be possible. However there are plans to incorporate Offender Assessment information into analyses which may help in this regard.
The second session was a thought-provoking panel debate that explored some of the issues raised by the Justice Data Lab. Panel members included Prof Fergus McNeill from the University of Glasgow, who eloquently summed up some of the limitations of the data labs approach. The detail is critical – provide wrong or unreliable data and the results will be too. Compiling a matched comparison group can only be done on a limited number of observed variables; unobserved variables may also exert an influence on outcomes. And reconviction (which is what we are looking at here) is a poor outcome variable – it is not an indicator of behaviour change but is produced by a range of interventions and actions interacting in complex ways.
Charlotte Weinberg, chief executive of Safe Ground, talked about the organisation’s experience of using the Data Lab and the usefulness of the insights provided by the Data Lab team beyond the basic comparison of reoffending. Having the skills to interpret and communicate the results are key, as is having robust enough data to submit in the first place. She stressed, and others on the panel agreed, that as valuable as the Data Lab resource is, it is just one of a range of sources of evidence and approaches to evaluation available, and it is important to draw on many rather than just one to get the full picture.