The problem
Users have told us that finding guidance for funding streams and conditions of grant on GOV.UK is difficult and time-consuming.
Many rely on Google to search for relevant information. But search results often return outdated pages, requiring users to verify information through forums and chats with other finance teams.
Linking directly to guidance is not a reliable solution, updates are often published as new pages rather than updating existing ones, and redirects are not always created. This can result in users accessing outdated guidance.
Exploring a chat-based solution
To address this problem, the team decided to test a chat feature, using ChatGPT to build a test model.
We created DfE Chat, training it to return summaries of the latest available guidance, with links to the official source pages for further details.
Initial testing within the team showed that the chat feature consistently provided results for the most current guidance. When asked for guidance from a date range in the past, the feature was able to return relevant historical information when available.
Considerations and potential issues
While the chat feature has shown promising results, there are challenges associated with using AI-driven chat tools for this purpose.
We identified the following concerns:
Accuracy and reliability
AI-generated responses depend on the training data and available sources. If guidance updates are delayed in the training data, users might receive outdated summaries.
There is a risk that ChatGPT may generate authoritative-sounding but incorrect or misleading responses.
Trust and verification
Users may develop a false sense of confidence in the responses provided by the chat tool without verifying information from official sources.
Including links to official guidance mitigates this risk, but users might still need to validate that the linked pages reflect the latest updates.
Handling policy and terminology changes
Policy language evolves, and AI models might not always capture nuances in funding conditions accurately.
Directing users to official documentation ensures they can see the precise wording used by DfE.
User expectations and adoption
Users may expect the chat tool to provide precise answers rather than summaries, leading to frustration if they still need to review full guidance documents.
Continuous user testing is essential to refine expectations and improve response quality.
Next steps: Testing and iteration
To evaluate the effectiveness of DfE Chat, we will:
- signpost the feature in the prototype as a ‘Chat’ option and test what users expect from a tool labeled this way
- monitor user interactions to assess if the chat meets their needs and whether it directs them to the correct sources
- observe how the chat function evolves as more real-world queries are fed into it, refining its ability to handle complex or ambiguous user requests
By iterating on this feature through user testing, we aim to determine whether a ChatGPT-powered tool can provide a more efficient and reliable way for finance teams to find up-to-date funding guidance.