Research: Quantification of the influence of Github Copilot on the productivity and happiness of the developer (2024)

Every day we use tools and form habits to achieve more with less.In 2021 our hypothesis was that it would improve the productivity of the developer and share the early user reports it did.We first had to fight with the question: what does it mean to be productive?

Why is the productivity of the developer so difficult to measure?

When it comes to measuring the productivity of the developer, there is little consensus and there are many more questions than answers.

• What are the "real" productivity measurements?1,,2]
• How valuable are self -requests of productivity?3]
• Is the traditional representation of productivity - output of input - a good fit for the complex problem solving and creativity involved in development work?4].

In a 2021 study we have established that the own image of developers has a turn of productivity - it is more related to havinga good day.

This is also not a disposable fraud.5] and that satisfied developers manage better [6,,7].

What do we think of the productivity of the developer at Github?

Because AI-assisted development is a relatively new field that scientists have to draw earlier.In the scale of their experience with the help of Github Copilot.We have designed our research method with three points in mind:

• Look holistically at productivity.At Github we like to thinkLargely and durableAbout the productivity of developers and the many factors that influence this.We usedSpace Productivity FrameworkTo choose which aspects to investigate.
• Record the first perspective of developers.We have completed various research rounds, including qualitative (perceptual) and quantitative (observed) data to collect the full image.We wanted to verify: (A) confirms the actual experiences of users what we broadcast from telemetry?To our large user base?
• Assess the effects of Github Copilot in daily development scenarios.When we set up our studies, we became extra careful to recruit professional developers and design tests on typical tasks that a developer could continue on a certain day.

Let's dig up and see what we have found!

Find 1: productivity of the developer goes beyond the speed

Through a large -scale study we wanted to see whether developers who use Github Copilot see benefits in other areas, in addition to accelerating tasks.

• Improving the satisfaction of developers.Between 60-75% of users reported that they feel more filled with their job, feel less frustrated during coding and can concentrate on more satisfying work when using Github Copilot.
• Retain mental energy.Developers reported that Github Copilot helped them to stay in the power (73%) and to retain mental efforts during repeated tasks (87%).day, and that certain types of work outlet [8,,9].

Table: Research response that measures the development of the productivity of the developer when using Github Copilot

All questions are modeled bySpace frame.

Developers see Github Copilot as a productivity support, but there is more to it than that user described the overall experience:

That was taken from our qualitative studyTo make Github Copilot shouldered the boring and repeated development work, the cognitive tension reduced.This makes room for developers to enjoy the more meaningful work that requires complex, critical thinking and problem solving, which leads to more happiness and satisfaction.

Find 2: ... but speed is also important

In the study we saw that developers reported that they perform tasks faster when they used Github Copilot, mainly repeated.Overlhelming observed that Github Copilot helps them perform tasks faster - can we observe and measure this effect in practice?Before we have carried out a controlled experiment.

Figure: Summary of the experiment process and the results

We have recruited 95 professional developers, they randomly divided into two groups and Timed, how long it took to write an HTTP server in Javascript.As we could - all developers were already familiar with JavaScript, we all gave the same instructions and we used the Github classroom to automatically score submissions for correctness and completeness with a test hell.Our experiment!

In the experiment we measured on average how successful each group was about to end the task and how long each group lasted to end.

• The group that Github Copilot used had oneHigher rate to complete the task(78% compared to 70% in the group without Copilot).
• The striking difference was thatDevelopers who used GitHub Copilot completed the task considerably faster - 55% faster than developers who did not use Github Copilot.Take the developers who used Github Copilot, on average needed 1 hour and 11 minutes to complete the task, while the developers who did not use Github Copilot on average 2 hours and 41 minutes.Diss results are statistically significant (P = .0017) and 95%confidence interval for the percentage of speed increase is [21%, 89%].

There is more to reveal!We carry out more experiments and a more thorough analysis of the experimental data that we have already collected - look at heterogeneous effects or potential effects on the quality of the code - and we plan extra academic publications to share our findings.

What do these findings mean for developers?

We are here to support developers while building software - including working more effectively and finding more satisfaction in their work.Github Copilot supports faster completion times, retains the mental energy of developers, helps them to concentrate on more satisfying work and ultimately to find more fun in the coding they do.

We also hear that these benefits become equipment for technical leaders in companies that have carried out early tests with Github Copilot.bring that joy.

With the arrival of Github Copilot, we are not only in exploring the impact of AI-driven code end tools!An evaluation with 24 studentsInGoogle's internal assessment of the end of the ML improved codeMore wide, the research community tries to understand the implications of Github Copilot in a number of contexts:education,,security,,Laboratoryas well as developerexerciseInbehaviourWe all teach everyone at the moment by trying Github Copilot in different settings.

Acknowledgment

We are very grateful to all developers who participated in the study and experiments - we would be in the dark without your input!GitHub nextthe experiment completed in collaboration with Microsoft Office of the Chief Economist and specifically in collaboration withSida PengInFoundation Cannon.