You are here

  1. Home
  2. Not-delivering-what-the-question-asked-for

Not-delivering-what-the-question-asked-for

Similar to the basic mistakes of the previous section, Generative AI output does not always follow the guidance and common-sense expectations when constraints are placed on the way questions are expected to be answered.

Not answering the question set / answering a different question to the one set

In this example, the answers given vary from some element of the core requirements set out in the question. The answer may use supporting elements from the question but then fail to meet the key request. It sometimes feels as if the student has replaced the question focus with another question but kept the setting or context.

One thing many LLMs have been trained on is not to offer an opinion; so, they struggle to offer useful insights when questions expect an opinionated answer, where the student is meant to take one side or the other (although they do work well in compare and contrast questions).

Example: The question said, do it; don't define it.

Question 

What is the mean of the set of numbers [23, 45, 67, 23, 43, 16, 35, 75]?

Answer

To find the mean (average) of a set of numbers, you add up all the numbers and then divide by the total count of numbers.

Commentary

The LLM has not given the numeric answer expected, but has instead defined the term ’mean’.

Answering a question in a manner different from that taught in the module material

While students can, and do, often read external content not given in the module materials and bring techniques and situations from previous study and work experiences, most modules are written with certain approaches and techniques that are expected to be reflected in the assessment. The module team and marking guides will indicate if deviations from the module approaches are permitted and how they should be handled. 

However, GenAI does not remain within the limits of the module content, or the question context, and can bring in material from elsewhere. Students who do not understand the output supplied by a GenAI tool can then incorporate widely different approaches from those expected on the course, including overly complex techniques to solve straightforward problems.

Not using the given scenario/case study/content given the question/course materials

You may find a full general answer that is not grounded in the context of the question, or that fails to draw on examples from supplied reading or referenced course content.

Excessive description and definitions in keywords in the context of the question/module

The submitted work covers a lot of detail about a wide range of issues that are marginally relevant to the question asked, or that look like a catalogue-list of all possible issues that might arise in a given example. 

You may be familiar with this as it sometimes happens for students in an exam situation where they have time and feel that a question is a good 'topic' for them, but they haven’t quite resolved what the question requires. They will often list everything they know about the topic and usually relying heavily on key definitions. 

Current GenAI tools similarly produce outputs where they will give overly complete lists, depth of descriptions and definitions, regardless of the expected depth and focus from the question.

Example: describe, describe, describe

Question

What types of presentation would be appropriate to deliver academic research findings?

Answer

Various presentation formats are suitable for presenting academic research findings, including:

  1. Conference Presentations: Oral presentations at academic conferences, often accompanied by slides summarizing the research methodology, results, and conclusions.
  2. Poster Presentations: Visual displays of research findings, presented at conferences or symposiums, allowing for interactive discussions with viewers.
  3. Thesis Defenses: Formal oral defenses of a thesis or dissertation before a committee, involving detailed explanations and defenses of the research.
  4. Seminars or Lectures: Formal lectures or seminars delivered to academic audiences, providing in-depth insights into the research and its implications.
  5. Panel Discussions: Participating in panel discussions alongside other researchers, offering diverse perspectives on a specific topic or research area.
  6. Webinars or Online Presentations: Virtual presentations delivered through online platforms, allowing for widespread dissemination of research findings to a global audience.
  7. Journal Club Presentations: Sharing research findings during journal club meetings, where participants critically analyze and discuss recent publications.
  8. Workshops or Training Sessions: Interactive sessions where researchers present findings and engage participants in discussions or activities related to the research topic.
  9. Teaching Sessions: Incorporating research findings into teaching materials or presentations for academic courses, workshops, or seminars.

The choice of presentation format depends on factors such as the audience, the nature of the research, and the goals of dissemination.

Commentary

  • That’s a long list, very thorough and pretty much complete.
  • A few of those descriptions are outside the coverage of the course materials.

 

Answering a question with a length that is entirely inappropriate for the number of marks available

Where a question is clearly looking for a short answer, the supplied answer is voluminous, going into extended details, descriptions or explanations.  Similarly, the converse can occur when a high mark question gets a very abbreviated answer.

Not working to word counts

Some GenAIs are generally poor at writing to a word count.  Either overlong answers to tight word deadlines (100+ words when set a 50-word limit) or short answers to more generous targets (<700 words given when a 1000 word limit is set, for example.)  Students who do not pay attention to the length of the output generated simply copy the text across.

In addition, if you ask for the word count to be reported. GenAI output will often show a mis-match between the stated word count and the actual word count. GenAI tools don't know how to count so they mis-report (hallucinate) a number for the reported length of the output text.

 

 

Back to main page