A key decision in beta was whether the service should be based around searching for a qualification, or whether filtering the possible results through asking questions would give better results for users.
Given that users are already able to check whether a qualification is approved by reviewing the Early Years Qualifications List (EYQL) spreadsheet, in the alpha phase we could conduct research on users’ existing mental models and how they get to an answer when checking qualifications in their settings.
Our aim was to find a design for the service that helped users to:
- find the correct qualification
- be confident that the information is complete
- avoid missing important information, or small variations between qualifications
- reach an informed decision on whether the qualification can be considered ‘full and relevant’
Our first interactive prototypes used a search-based approach, which seemed to match users’ existing behaviour. However, user research sessions where we asked users to check sample qualifications with the search-based approach showed that there were significant challenges.
Design challenges
The design challenges we faced included:
- A service already exists, so we need to be aware of our users’ pre-existing processes, behaviours and expectations.
- Qualifications are categorised by the awarding organisation, not under the college or university where the practitioner studied. This can make it difficult for managers to extract the information they need from a CV. For example, some users entered the name of the college where the qualification had been studied, rather than the awarding organisation.
- Presenting qualification information individually rather than all together on a spreadsheet creates some specific design challenges. For example, when users can see all the full and relevant qualifications, it may be easier to understand what information on a CV is relevant.
- The combination of a technically limited search box and a lack of guidance and structure around what users were entering in the search box led to an excessively high rate of failure, where users were unable to find out whether the qualification was full and relevant. This reduced users’ confidence in the service; for example, is a qualification not showing in the service because it does not match what the user entered, or because it is not full and relevant?
Although users were telling us that they wanted a service based around a search box, we were concerned that users were not consistently able to get to an accurate result. We ran design ideation sessions to work on the problem and developed an alternative journey which was question-based.
Designing a question-based approach
This approach asked a single question per page, narrowing down the number of matching qualifications, before presenting users with an as small as possible number of qualifications to choose from.
A question-based journey also allowed us to:
- signpost users searching for qualifications which are excluded from the EYQL to more appropriate guidance
- be specific about the information we wanted users to enter
- provide contextual hint text and guidance to help users to find the information needed to get an accurate result.
The questions we added were:
‘Where was the qualification awarded?’ – this allowed us to signpost users wanting to check qualifications achieved outside the UK to more appropriate resources.
‘When was the qualification started?’ – this question allows us to significantly filter the number of qualifications returned on the search results page.
‘What level is the qualification?’ – this question allows us to:
- filter down the number of qualifications returned on the search results page
- signpost users looking for a level 2 qualification achieved between 2014 and 2019 to the Ofqual website, as these qualifications are not listed on the EYQL
‘What is the awarding organisation’ – we were able to use data from the EYQL to present a filtered list of awarding organisations that matched the user’s previous answers.
A/B testing the two search methods
In the early stages of beta, we were primarily A/B testing the two search methods. We again tested with early years managers and gave them a mocked-up CV. We showed users both versions of the service, evaluated which version was more successful and asked the participants which version they preferred.
User research participants had different levels of experience with the EYQL.
Some of our users are very familiar with the EYQL. Our assumption was that these users know what they are looking for so may prefer the search-based journey.
Some of our users may be less experienced managers who are not particularly familiar with the EYQL. Our assumption was that the question-based approach flow might be a better fit for them as it does more to guide them through the journey.
Research outputs
User research indicated that almost all participants were far more successful in reaching the correct outcome going through the question-based journey.
While some more users preferred the search-based journey, it was again not reliable enough in getting users to an accurate result. It allowed more room for error as it narrowed down the results by searching for keywords of the qualification name. It proved difficult for users to pinpoint the correct qualification as there are qualifications with similar wording, or with the same name but a different awarding organisation.
Next steps
The team has moved forward with the question-based journey. We will continue iterating the questions to make sure that they are clear to users and allow us to filter as much as possible. We will also continue to make sure that we are supporting users of all experience levels to get to an accurate result.
The question-based journey was significantly more successful for users of all experience levels. Some users preferred this journey, but even users who preferred the search-based version of the service were happy to use this version. No user research participants mentioned that there were too many questions or that it would take too long to get to the result.