Prompt Mastery Part III: Chain-of-Thought and Self-Consistency prompting

As businesses increasingly rely on large language models (LLMs) to handle complex tasks, ensuring accuracy and consistency in outputs is crucial. Two advanced prompting techniques—Chain-of-Thought (CoT) prompting and Self-Consistency prompting—are powerful tools for improving an LLM’s reliability, particularly in applications like customer support, text summarisation, calculations and detailed processes.

In this blog, we’ll explore how these techniques work and how they can benefit your business, from process documentation to copywriting. Whether you’re automating multi-step processes or need highly consistent answers, these approaches can help deliver results you can count on.

What is Chain-of-Thought Prompting?

Chain-of-Thought (CoT) prompting guides an LLM to break down its response into logical steps, letting it “think aloud.” This step-by-step structure is especially useful for complex calculations, multi-step instructions, and tasks that require clear reasoning. By prompting the LLM to work through each part of the task, CoT prompting reduces the chance of errors and provides a transparent path to the answer.

Example: Suppose you want an LLM to calculate the total cost for a customer’s purchase, including a discount and tax. Instead of asking for a final answer, CoT prompting allows you to see each step.

Prompt with Chain-of-Thought:

Calculate the final cost of a £200 purchase with a 10% discount and 20% VAT. Calculate and show the output step-by-step.

Output:

This prompt style makes each part of the calculation transparent and allows you to confirm the reasoning.

1. Original price: £200 
2. 10% discount: £200 * 0.10 = £20 
3. Discounted price: £200 - £20 = £180 
4. VAT (20%): £180 * 0.20 = £36 
5. Final cost: £180 + £36 = £216

Business Applications for CoT Prompting:

Financial Analysis: Breaking down calculations for tax, profit margins, or budgets.
Customer Service: Providing clear troubleshooting instructions with step-by-step guidance.
Project Management: Outlining sequential tasks for complex workflows.

Understanding Self-Consistency Prompting

Self-Consistency prompting is a technique that improves accuracy by internally generating multiple responses to the same prompt, then comparing those responses to find the most consistent answer. This was first proposed in Wang et al. (2022).

This method allows the LLM to “double-check” its work by generating variations and analysing commonalities, which makes it especially effective for tasks with nuanced or potentially ambiguous responses.

Here’s how it works: When you provide a prompt, the LLM can generate multiple outputs based on its probability-based reasoning. Each output represents a possible answer. By repeating the prompt multiple times, Self-Consistency prompting helps identify the response pattern that appears most frequently, making it the most likely “correct” answer. Ideally, you'd try the prompt 3-5 times, collate these answers and present these back to the model; with a task to determine which is the best.

Example: You need to make a process to reset your azure password.

Prompt for Self-Consistency:

Imagine three experts with differing opinions are asked to provide this response:
How do you reset your azure password?

Responses:

Response 1: “Go to the Azure portal and click ‘Can’t access your account?’ Enter your work email address, complete the CAPTCHA, and follow the email instructions to reset your password.”

Response 2: “Visit the Azure portal, select ‘Can’t access your account?,’ then enter your work email and complete the CAPTCHA. Check your inbox for a reset email and follow the link to create a new password.”

Response 3: “On the Azure portal login page, click ‘Can’t access your account?’ Enter your work email, complete the CAPTCHA, and use the email link to reset your password.”

Most Consistent Output:

“Go to the Azure portal and click ‘Can’t access your account?’ Enter your work email, complete the CAPTCHA, and follow the link in your email to reset your password.”

Through Self-Consistency prompting, we arrive at a response that combines the most consistent instructions, ensuring clarity and reducing ambiguity.

After comparing multiple responses, the model converges on the most common instructions, which are typically clearer and more reliable.

Universal Self-Consistency

Universal Self-Consistency is a more advanced prompting technique that enhances accuracy by evaluating the LLM's consistency across multiple, slightly varied versions of the same prompt. By testing the model’s response to different phrasing or formats of a question, Universal Self-Consistency ensures a more robust and reliable output. This technique is especially valuable for tasks requiring high precision, as it confirms that the LLM can consistently provide the correct answer, regardless of subtle prompt differences.

Why Self-Consistency Works Well with LLMs

LLMs generate responses based on probabilities, meaning there can be slight variations each time you prompt them. Self-Consistency prompting “smooths out” these variations by focusing on repeated patterns across multiple outputs. By finding and selecting the most consistent elements, this technique yields a response that aligns more closely with the intended instruction. Combining this with one or few-shot prompting, you can optimise your outputs further.

Business Applications for Self-Consistency Prompting:

Customer Service FAQs: Ensuring consistent, reliable answers for frequently asked questions.
Data Summaries: Producing uniform summaries of reports or recurring data patterns.
Standardised Content: Creating descriptions or instructions that are consistently clear.

Combining Chain-of-Thought and Self-Consistency Prompting

Combining self-consistency with chain-of-thought prompt engineering

Combining CoT and Self-Consistency prompting can take the reliability of LLM outputs even further. CoT guides the model to work through each step logically, while Self-Consistency double-checks the results by filtering out discrepancies. For tasks that require step-by-step reasoning and accuracy, this combination is powerful.

Example Combination Prompt (run 3 times):

Calculate the total cost of a £150 order with a 5% discount and 18% VAT, showing each calculation step.

Output (1st stage):

Response 1:
Original price: £150
5% discount: £150 * 0.05 = £7.50
Discounted price: £150 - £7.50 = £142.50
VAT (18%): £142.50 * 0.18 = £25.65
Final cost: £142.50 + £25.65 = £168.15

Response 2:
Original price: £150
Discount amount: £150 * 0.05 = £7.50
Price after discount: £150 - £7.50 = £142.50
VAT (18%): £142.50 * 0.18 = £25.65
Total cost: £142.50 + £25.65 = £168.15

Response 3:
Original price: £150
5% discount applied: £150 * 0.05 = £7.50
Price after discount: £150 - £7.50 = £142.50
VAT added: £142.50 * 0.18 = £25.65
Total cost after VAT: £142.50 + £25.65 = £168.15

Choosing the best output- intermediate prompt:

Choose the most consistent answer from the following and only return the correct answer:
```previous answers enclosed in triple quotes```

Final output:

The total cost is £168.15, calculated as follows: Start with the original price of £150, apply a 5% discount (£7.50), resulting in £142.50, then add 18% VAT (£25.65) to get the final cost of £168.15.

In this example, Self-Consistency helps verify the Chain-of-Thought calculations. By cross-referencing each output, you can be confident that £168.15 is accurate. This approach is particularly useful for financial reporting, inventory tracking, and other complex business applications where reliability is essential.

Automate, optimise, succeed,

Whilst this may seem labour-intensive, by triggering workflows in make.com, zapier, n8n or similar, businesses can utilize these prompt patterns in a consistent, templated fashion. Chaining together a set of prompt input/outputs especially from different models (and different companies providing the model) can provide an extremely powerful 'mixture of experts' approach to automating business processes and providing optimal outputs.

Multi-step workflow for chain-of-thought and self-consistency prompting — Using automation tools, you can leverage multi-stage workflows easily

Bringing Accuracy and Consistency to Business Applications

Chain-of-Thought and Self-Consistency prompting are highly effective ways to enhance the reliability of LLMs in business contexts. By guiding the model through a logical sequence with CoT and verifying outputs through Self-Consistency, these methods ensure accuracy in tasks like financial analysis, data summaries, and customer service responses.

At Colne Data & AI, we specialise in implementing these advanced prompting techniques on the latest LLMs within automations and integrations that deliver reliable, accurate results for your business. Our team is here to help you design AI solutions that fit your unique needs, streamlining your processes with precision and efficiency.