BIS field study show AI use can boost productivity among programmers. Still, the impact is less pronounced on senior employees.

A Bank for International Settlements’ field study on AI use by Ant Group programmers demontrated a clear boost in the number of lines of code produced by staff exposed to the Large Language Model. However, productivity gains are statistically significant only among entry-level or junior staff, while the impact on more senior employees is less pronounced. 

Generative artificial intelligence (Gen AI) tools hold significant promise for enhancing worker productivity across various fields. These AI models have demonstrated capabilities comparable to humans in areas like clinical care, education, language modelling, art, music and design. A growing body of literature explores commercial and non-commercial applications, ethical considerations, regulatory frameworks, and implications for security and education. However, empirical research on AI’s impact on productivity in tasks requiring cognitive abilities remains scarce.

Contribution

„We investigate the impact of Gen AI on labour productivity through a field experiment in the coding industry. In September 2023, Ant Group launched CodeFuse, a large language model (LLM) designed to assist programming teams. In our experiment, one group of programmers had access to CodeFuse (the treatment group), while another group did not (the control group). By comparing similar employees from these two groups, we assessed how AI affected their productivity.” – the authors said.

Findings

„Our findings indicate that LLMs can significantly boost productivity among programmers. Productivity (measured by the number of lines of code produced) increased by 55% for the group using the LLM. Approximately one third of this increase was directly attributable to code generated by the LLM. The remaining productivity gains were likely due to improved efficiency in other coding tasks, as programmers had more time available.

However, the productivity gains were statistically significant primarily among junior staff, with a less pronounced effect on senior employees. This difference appears to stem from lower engagement with the LLM by senior programmers, rather than the tool being less useful to them. The rate at which programmers accepted the LLM’s suggestions did not vary with experience level, suggesting that the lower impact on senior programmers’ productivity was due to less frequent use of the tool.” – the authors concluded.

More details:

BIS Working Papers No 1208 – Generative AI and labour productivity: a field experiment on coding
by Leonardo Gambacorta, Han Qiu, Shuo Shan and Daniel M Rees