{"id":1273,"date":"2023-09-06T14:05:15","date_gmt":"2023-09-06T14:05:15","guid":{"rendered":"https:\/\/ignos.blog\/mathematical-skills-in-llms"},"modified":"2023-09-06T14:24:32","modified_gmt":"2023-09-06T14:24:32","slug":"mathematical-skills-in-llms","status":"publish","type":"post","link":"https:\/\/ignos.blog\/en\/mathematical-skills-in-llms","title":{"rendered":"Mathematical skills in LLMs"},"content":{"rendered":"\n

Mathematical Reasoning in LLMs<\/h2>\n\n

Large Language Models (LLM) have found a niche in a multitude of tasks related to the analysis and generation of content in different areas of knowledge. In the last decade, the use of LLMs has been expanding from its beginnings mainly associated with text generation and translation, to the current moment where its applications have become numerous.<\/p>\n\n

These models, trained with large volumes of information in text format, learn linguistic and grammatical patterns, which allows them to answer questions, create content, and assist in data analysis<\/a> tasks. However, since they are based on probabilistic mechanisms, these responses, contents, and analyses may lack precision in areas that require exact, not approximate, solutions, such as in the case of mathematical problems. In this way, the questionable mathematical calculation and reasoning abilities in most models have led to their use in scientific or data processing environments not spreading as rapidly as in tasks based on content creation.<\/p>\n\n

In the following sections, we will attempt to assess the mathematical skills of some of the most commonly used models today and explore the potential for improvement through the utilization of certain strategies.<\/p>\n\n

Evaluated Models<\/h2>\n\n

In order to assess the mathematical capabilities of widely used LLMs, we have chosen three models that demonstrate high performance in generic reasoning tasks. <\/p>\n

\n
\"\"<\/figure><\/div>\n

Firstly, GPT-4<\/strong>, the latest iteration of the GPT series, known for its deep neural network architecture and its ability to process and generate text with advanced semantic comprehension. Its training, based on extensive datasets, enables it to provide more precise and contextually relevant responses.<\/p>\n\n

Similarly, LLama2<\/a><\/strong> is emerging as a significant innovation in the field, as it distinguishes itself by being distributed under the concept of open-source software. Its most comprehensive version, an LLM with 70 billion parameters, has managed to achieve performance levels comparable to established proprietary models.<\/p>\n\n

Finally, the Bard<\/strong> model stands out for its specialized focus on generating coherent narratives; it is capable of creating texts that not only adhere to grammatical norms but also capture the emotional essence of a narrative. Currently, supporting Bard is the Palm 2<\/strong> model, which, in its latest version, contributes enhanced mathematical training, among other capabilities.<\/p>\n\n

Mathematical Abilities<\/h2>\n\n

To assess the overall performance of the different LLMs in a mathematical problem-solving environment, a set of prompts in English covering various mathematical areas is proposed. These prompts have been generated using the “Mathematics Dataset”<\/strong> library, designed to evaluate the mathematical abilities of learning models. The following tests have been conducted without prior context:<\/p>\n\n

Id<\/th>Problem<\/th>Prompt<\/th>Answer<\/th><\/tr><\/thead>
1<\/td>Arithmetic Addition and Subtraction<\/td>Evaluate -11 – (-21 + 16) – (-1 + 12)<\/td>-17<\/td><\/tr>
2<\/td>Prime numbers<\/td>Is 18001811 a prime number?<\/td>True<\/td><\/tr>
3<\/td>Linear Algebra<\/td>Solve -388h + 131 = -373<\/em>h + 416 for h<\/td>-19<\/td><\/tr>
4<\/td>Algebra Polynomial Roots<\/td>Suppose -5t2 – 72845<\/em>t – 291300 = 0. Calculate t<\/td>-14565, -4<\/td><\/tr>
5<\/td>Differential Calculus<\/td>Differentiate -8893y4 + 2<\/strong><\/em>y<\/strong>2 + 8545 with respect to y<\/td>-35572y3 + 4<\/em>y<\/td><\/tr>
6<\/td>Polynomials<\/td>Let k(p) = -31*p + 127. What is k(4)?<\/td>3<\/td><\/tr><\/tbody><\/table><\/figure>\n

The results for each of the tests are:<\/p>\n\n

Id<\/th>GPT4<\/th>Bard<\/th>LLama2<\/th><\/tr><\/thead>
1<\/td>-17<\/td>-17<\/td>5<\/td><\/tr>
2<\/td>No answer<\/td>True<\/td>False<\/td><\/tr>
3<\/td>-19<\/td>19<\/td>-19<\/td><\/tr>
4<\/td>-14565, -4<\/td>-783.93, 783.93<\/td>-14569, 0<\/td><\/tr>
5<\/td>\u221235572y<\/em>3 + 4y<\/em><\/td>-35572y3 + 4y<\/td>-8893y4 + 2y2 + 854<\/td><\/tr>
6<\/td>3<\/td>3<\/td>3<\/td><\/tr><\/tbody><\/table><\/figure>\n

From the previous results, it can be observed that GPT-4 achieves a notable level of accuracy, correctly solving 5 out of the 6 problems proposed, while at the other end, LLama2 only manages to solve 2 out of the 6 problems.<\/p>\n

\n
\"\"<\/figure><\/div>\n

Alternatives and Improvements<\/h2>\n\n

Based on the conducted tests, it appears that if one were to seek valid solutions to mathematical problems of any type, none of the analyzed LLMs would be a valid choice, as they were unable to solve all of the test cases. However, alternative approaches have been proposed when formulating the resolution of mathematical prompts that significantly improve the results, for example:<\/p>\n\n