Evaluation using PROMPT

Young female student looking at books in book shop and surrounded by lots of books
The ability to critically evaluate information is an essential research skill. It is particularly important when you are producing literature reviews and need to critically appraise and analyse the literature.

The PROMPT mnemonic (Presentation, Relevance, Objectivity, Method, Provenance, Timeliness) provides a structured approach to critical evaluation of information.

Presentation

The presentation of information affects how we receive and perceive it. Poor use of language can make it difficult to understand the arguments. A poor structure and confusing layout can make it hard to navigate.

Examples of poor presentation:

a confusing layout, illogical structure, or no obvious structure at all
mistakes in spelling, grammar, or typographical errors
an inappropriate or ineffectual writing style
poor reproduction.

However, try not to let poor presentation stop you from using what might otherwise be high quality, relevant information. Look beyond the appearance and consider other PROMPT criteria to fully assess the quality of the content.

Relevance

To consider the relevance of a piece of information, you need to assess whether it meets your needs. Even if a source provides high-quality information, it may not be relevant to the question you are asking or the scope of your research.

Be clear about your requirements.

Learn to skim read and/or scan information to get a quick idea of what it is about. This will help you avoid wasting time reading irrelevant information. Look at the title, abstract or summary, keywords, and descriptors. For more details on reading techniques see the Reading efficiently pages within OU skills for study.

Things to consider:

Focus: it may have a relevant title but focus on an aspect which is not relevant to you.
Level: is it too detailed/specialised or too general/simple for your needs?
Geography: does it relate to countries or areas which you are not interested in?
Time period: it may relate to your topic but address it within a different period.

Objectivity

In an ideal world, information sources would present all the evidence and arguments and leave you to draw your own conclusions. In the real world, all information comes from a position of interest, although this may not be intentional. Objectivity, therefore, may be an unachievable ideal.

This means that it is up to you, the reader, to identify the positions represented in what you read, and to take account of this when you interpret the information. Remember that your own belief systems and opinions will also influence your reading.

Sometimes authors explicitly express a particular viewpoint. This is perfectly valid if they are clear about the perspective they represent. Hidden bias or errors of omission can be misleading.

When producing a literature review, it is important that you recognise any selective interpretation of data and comment on any significant omissions or biases that you come across in other people’s research.

Things to watch out for:

Perspectives: Do the authors state clearly the viewpoint they are taking? Are diverse perspectives presented?
Unconscious bias and excluded voices: What do you know about the identity of the author(s)? Considering your research as a whole, have you included diverse voices, perspectives, and experiences (e.g. from different countries including the Global South)? Are you selecting sources which confirm your own biases or seeking a diverse range of perspectives on an issue?
Opinions: Some sources present unsubstantiated theories for debate. Look out for opinion presented as fact.
Language: Look out for language that is either emotionally charged or vague.
Sponsorship: Sometimes research is funded by a sponsoring body. For example, academic research may be sponsored by industry or government. This does not necessarily make the research less objective, but it may make its interpretation selective. Make sure that all vested interests are clearly identified and that the sponsors are happy to give access to the actual research data.

Method

For this aspect of PROMPT we do not refer to the evaluation of research methods themselves, but to the information produced as a result of using particular methods.

Do not assume that because research has been published, its methodology is rigorous or inclusive.

Things to consider:

Is it clear how the research was carried out?
Were the methods appropriate? Do researchers address any differences in outcomes between groups (e.g. between different age groups, ethnicities, or genders)?
Critically question the research methods. Think about sample size and nature, use of control groups, questionnaire design. For example, was the sample used representative? Does it include diverse characteristics?
If the information discusses specific communities, are first-person experiences and views of members of that community considered?
Are the results produced consistent with the methods stated?
Are the methods suitable for your needs? Do you need the methods to be the same as, or different, to your own?

Provenance

The provenance of a piece of information (i.e., who wrote and produced it) can help you assess its reliability. It is, therefore, important to be able to identify the author, sponsoring body or source of your information.

Knowing about the identity of an author or publisher will help you evaluate whether your research includes diverse voices and perspectives.

Knowing about the sponsoring organisation helps you understand what their main 'business' is (e.g., commerical, voluntary, research), how well established it is, who the people involved are, and who they are linked with.

Knowing how information is published can help you identify how reliable it is. For example, has it been through an editorial or peer review process?

Provenance can therefore be an indirect clue to the reliability of information – a safety net that gives you the opportunity to check things out. Provenance can also affect other people’s confidence in the sources you are citing.

Treat anonymous information with caution.

Things to consider:

Authors

Are they acknowledged experts in the subject area? Are they respected and reliable sources?
Are their views controversial? Have they been frequently cited by other authors? (To find out whether material has been frequently cited requires either prior knowledge or a citation search).
Are they known to have a particular perspective on the topic?

Sponsoring organisations

What type of organisation is it? For example, is it a commercial company, voluntary organisation, statutory body, research organisation?
How well established is the organisation?
Does the organisation have any vested interests in the subject area?

Publication methods

Publication methods vary between different types of information. Anyone can publish on the web or post to a discussion list, whereas journals and books are often more selective.

What do you know about the editor and/or the editorial board? How might editorial policy influence what is published? Remember that the publication practices of some commercial and academic publishers result in some authors being excluded from their publications.
Is the journal well regarded? Does it have a high rating in the Journal Citation Reports? Does this matter?
Is the information peer reviewed? Many electronic journals do not have a peer review process.

Timeliness

Consider the date when the information was produced or published. This can help you assess its quality and relevance. This is not as simple as saying that ‘good’ information must be up to date; it depends on your information need.