Measuring LLMs’ Ability to Perform Cryptanalysis

Researchers have developed a new benchmark to assess the ability of large language models to perform mathematical cryptanalysis. The benchmark, called CryptanalysisBench, tests whether these models can discover new attacks against historical algorithms. A model from Anthropic was found to have successfully identified new attacks, demonstrating its capabilities in this area. The benchmark focuses on cryptanalysis, which combines mathematical reasoning and cybersecurity, two areas where large language models have shown significant advancements. This benchmark provides a testbed for evaluating the limits of large language models in cryptanalysis. The goal is to understand how well these models can perform tasks that require both mathematical reasoning and cybersecurity expertise.