can you share a prompt to test llama with mmlu data ?

[dh12306]

I tried this prompt but didn't work well :

The following are multiple choice questions (with answers) about  abstract algebra.
========example====== 
In order to make the title of this discourse generally intelligible, I have translated the term "Protoplasm," which is the scientific name of the substance of which I am about to speak, by the words "the physical basis of life." I suppose that, to many, the idea that there is such a thing as a physical basis, or matter, of life may be novel—so widely spread is the conception of life as something which works through matter. … Thus the matter of life, so far as we know it (and we have no right to speculate on any other), breaks up, in consequence of that continual death which is the condition of its manifesting vitality, into carbonic acid, water, and nitrogenous compounds, which certainly possess no properties but those of ordinary matter.
Thomas Henry Huxley, "The Physical Basis of Life," 1868
From the passage, one may infer that Huxley argued that "life" was
----here are the choices,you need to select one answer,return the order number----:
['a force that works through matter', 'essentially a philosophical notion', 'merely a property of a certain kind of matter', 'a supernatural phenomenon']
Answer:2
========example====== 
Read the the following quotation to answer questions.
The various modes of worship which prevailed in the Roman world were all considered by the people as equally true; by the philosopher as equally false; and by the magistrate as equally useful.
Edward Gibbon, The Decline and Fall of the Roman Empire, 1776–1788
Gibbon's interpretation of the state of religious worship in ancient Rome could be summarized as
----here are the choices,you need to select one answer,return the order number----:
["In ancient Rome, religious worship was decentralized and tended to vary with one's social position.", 'In ancient Rome, religious worship was the source of much social tension and turmoil.', 'In ancient Rome, religious worship was homogeneous and highly centralized.', 'In ancient Rome, religious worship was revolutionized by the introduction of Christianity.']
Answer:0
========example====== 
The following quote is from Voltaire in response to the 1755 Lisbon earthquake.
My dear sir, nature is very cruel. One would find it hard to imagine how the laws of movement cause such frightful disasters in the best of possible worlds. A hundred thousand ants, our fellows, crushed all at once in our ant-hill, and half of them perishing, no doubt in unspeakable agony, beneath the wreckage from which they cannot be drawn. Families ruined all over Europe, the fortune of a hundred businessmen, your compatriots, swallowed up in the ruins of Lisbon. What a wretched gamble is the game of human life! What will the preachers say, especially if the palace of the Inquisition is still standing? I flatter myself that at least the reverend father inquisitors have been crushed like others. That ought to teach men not to persecute each other, for while a few holy scoundrels burn a few fanatics, the earth swallows up one and all.
—Voltaire, in a letter, 1755
The ideas expressed by Voltaire, above, best illustrate which of the following characteristics of Enlightenment intellectuals?
----here are the choices,you need to select one answer,return the order number----:
['Many were accomplished scientists, who added important pieces to human understanding of the universe.', 'They utilized new methods of communicating their ideas, such as salons and inexpensive printed pamphlets.', 'Most rejected religion altogether and adopted atheism as the only credo of a rational man.', 'Many believed that the new scientific discoveries justified a more tolerant and objective approach to social and cultural issues.']
Answer:3
==========answer this question below===========
{question}
----here are the choices,you need to select one answer,return the order number----:
{choices}
Answer:

then , flowed the demo code @jindongwang @madhavMathur :

dataset = pb.DatasetLoader.load_dataset("mmlu")
...

for prompt in prompts:
    preds = []
    labels = []
    for data in tqdm(dataset[5:]):
        # process input
        input_text = pb.InputProcess.basic_format(prompt, data)
        label = data['answer']
        raw_pred = model(input_text)
        print(raw_pred)
        break

[Immortalise]

Hi, here is an example prompt:

"Answer the following multiple-choice question about {task} by selecting the correct option: 'A', 'B', 'C', or 'D'. \nQuestion:{content}\nAnswer:",

You could find more example prompts for mmlu in promptbench/prompts/task_oriented.py or promptbench/prompts/role_oriented.py.