One Language, Many Gaps: Evaluating Dialect Fairness and Robustness of Large Language Models in Reasoning Tasks

What is ReDial?

ReDial (Reasoning with Dialect Queries) is a benchmark of more than 1.2K parallel Standardized English-AAVE query pairs.

ReDial benchmarks four canonical reasoning tasks, namely algorithm, logic, math, and comprehensive reasoning. The task formulation is linguistically diverse, addresses cornerstone problems in human reasoning, and is of particular interest as it is challenging for LLMs.

How is ReDial constructed?

See a video demonstration of our workflow below:

To obtain a highly curated dataset, we sample from seven widely used and established benchmarks.

For Algorithm, we sample 164 instances from HumanEval, 150 from sanitized MBPP. For Math, we sample 150 instances from GSM8K and 150 from SVAMP. For Logic, we sample 200 instances from LogicBench, and 162 from Folio and Folio-perturbed. For Comprehensive reasoning, we sample 240 instances from AsyncHow.

We hire AAVE speakers and instruct them to rewrite each instance. For algorithm tasks that require an understanding of code to keep the semantics, we specifically hire expert AAVE annotators with computer science backgrounds.

To ensure the quality of the annotation, we conduct careful validations to ensure its naturalness and correctness.

After this process, we obtain a high-quality, human-annotated dataset ReDial with more than 1.2K Standard English-AAVE parallel prompts in four canonical reasoning tasks. ReDial is the first benchmark of its kind and enables easy testing and analysis of LLMs’ dialect fairness and robustness in reasoning tasks.

Quick start

Go to this paper’s github repo and follow the ReadMe to have a quick start! The evaluation is fairly easy! You can either follow the instructions in this paper’s repo or just use the standard pipelines as you want as we only modified example inputs but the outputs are exactly the same as source datasets.

Our findings

Below are the pass rates for testing models with zero-shot and CoT prompting on ReDial. Results in bold show a statistically significant deviation between AAVE and Standardized ReDial (i.e., models have significant drops in AAVE).

And here is the table for task-wise performance breakdown (results are averaged across all models).

We find that all models and all tasks have drops in AAVE data! This indicates that our benchmark poses huge challenges to models, both in terms of absolute performance and with respect to their dialect robustness and fairness.

Other findings

We also find that the lack of understanding of AAVE does not explain the performance drop, and that the drop cannot be easily mitigated by advanced prompting method: this indicates that even if non-standard dialect users are willing to pay more at inference time, they still cannot get as good service as standard dialect users!

You can see more details in our paper and also try it out yourself!

BibTeX citation

If you find our work useful, you can cite it as:

    @article{lin2024one,
  title={One Language, Many Gaps: Evaluating Dialect Fairness and Robustness of Large Language Models in Reasoning Tasks},
  author={Lin, Fangru and Mao, Shaoguang and La Malfa, Emanuele and Hofmann, Valentin and de Wynter, Adrian and Yao, Jing and Chen, Si-Qing and Wooldridge, Michael and Wei, Furu},
  journal={arXiv preprint arXiv:2410.11005},
  year={2024}
}