Enabling Copilot: The Journey

How it started

It started off at Plato Symposium (a gathering for engineering leaders) where I heard other engineers talk about their experience with the tool. Intrigued by their discussions, I decided to try out the free trial over the weekend. In short, the tool blew my mind. The next step was to convince the stakeholders at our company that we need to evaluate Copilot.

Stakeholders

Stakeholder 1: Engineering leadership

Once I became convinced of the potential of this tool, I proceeded to create a brief demonstration, which I then shared with our engineering leadership via Slack. This action initiated a discussion and key individuals had valuable insights: the head of engineering had already utilized ChatGPT to write code, our most senior engineer had access to Copilot for free due to their open source contributions and was using it, and the head of product had experimented with Copilot and found it impressive. With a consensus emerging among the leadership, we agreed that this tool had significant potential and warranted further evaluation.

Concern: Licensing

The first concern that came up was licensing, specifically whether we would breach open source licensing terms if we used the tool.

Resolution / Information

• GitHub has 2 ways they solve for this
  • They let you block code that matches public code in the settings
  • Section 6 in their agreement offers an indemnity clause that protects you from legal action
• There are complementary tools like blackduck that can help you with attributing or restricting use of open source code if you want extra protection.
• Amazon CodeWhisperer claims to tackle this concern by training their model on only permissible code and internal repositories in case you are looking at their solution.

Concern: Budget

We had not planned for this tool during the financial planning phase and wanted to get a sense of how much it would cost us.

Resolution

• GitHub was willing to offer the tool for free for 30 days. This gave us a way to evaluate the tool before we start getting charged.
• We looked at how many engineers had made a commit in the past 60 days. This gave us a sense of the max licenses we would need.
• Estimated cost vs value: To keep things very basic, if an engineer gets paid 150,000$ per year, it needs to save ~16 minutes per month to be cost-effective purely from a time perspective. This was a bet we were willing to take.
  • Assuming a salary of 150,000$ per year, the cost per minute is 150000/(52*40*60) = ~1.2$ per minute. Copilot costs 19$ per month per user. So it needs to save 19/1.2 = ~16 minutes per month to be cost-effective.

Stakeholder 2: Security

The next stakeholder was security.

Concern: Sensitive information

We had to make sure that we were not exposing any sensitive information via the tool and protecting our IP.

Resolution

• Information on data collected by Copilot. It was important that they are not going to use our code to train their models.
• Additional information provided by GitHub for enterprise accounts/business license
  • User edit actions, source code snippets, and URLs of repositories and file paths are sensitive data. Consequently, several measures of protection are applied, including:
    • The transmitted data is encrypted both in transit and at rest; Copilot-related data is encrypted in transit using TLS, and at rest using Microsoft Azure’s data encryption (FIPS Publication 140-2 standards).
    • Access is strictly controlled. The data can only be accessed by (1) named GitHub personnel working on the GitHub Copilot team or on the GitHub platform health team and (2) Microsoft personnel working on or with the GitHub Copilot team.
    • Role-based access controls and multi-factor authentication are required for personnel accessing Prompt or Suggestion data.
• With the amount of attention being paid to GenAI, We are also trusting the security research community to find vulnerabilities in the tool and report them to GitHub.
  • We will also probably try to set up a separate repository and try to get Copilot to spit out company specific code by feeding it prompts from our codebase at a later point in time.

Stakeholder 3: Legal

The next stakeholder was legal. It was kind of eye-opening to get into the "ifs" and "buts" when diving deeper into the terms of service.

Concern: Indemnity clause

Other than concerns around IP protection which I already covered under security, We had to make sure that we were not exposing ourselves to any legal risk by using the tool. Specifically

• Double indemnity. Section 6 that I had mentioned earlier also has a paragraph stating "Customer will defend GitHub and its Affiliates against any third-party claim.....". Did that expose us to legal bills?
• Also articles like this added to the questions.

Resolution

• We managed to resolve this by having an email thread with our GitHub account manager and legal team. At a high level, we got confirmation that
  • The concern from the article about "if your agreement with them happens to have one..." was resolved since we have the enterprise license.
  • Use of the tool would cover the entire use of the product as a whole.
  • We would use a filter to block open source code.

Deployment

The deployment process was pretty straightforward. Steps were

1. Send a message informing engineering that we are enabling the tool for them and that they can opt in if they want to.
   • Got ChatGPT to draft the message for me.
2. Enable Copilot.
   • Pay attention that the settings (eg: block open source code) are applied correctly when enabling Copilot.
3. Setup a team in GitHub such that each member of the team gets access to the tool.
   • This would allow us to distribute the load by creating administrators for the GitHub team, and they would have control over who gets access to Copilot.
   • This also gives us an easy way to track the number of licenses we are using.
4. Setup a Q&A session with GitHub customer success where they would demo the tool and answer any questions.
   • This was scheduled a week after we opened up the tool to engineering. This would offer early adopters the opportunity to experiment with the tool and bring forth specific questions.
   • Some of the tricks in the demo to improve the performance of the suggestions were useful.
5. Go over the usage stats with GitHub around the 25th day of the trial.
   • This was not very useful since they only had stats for Visual Studio Code users while we were a major JetBrains shop.
   • This increased the dependency on a survey to get feedback about the tool.
6. On the 26th day, send a survey to the adopters to get their feedback.
7. Around the 29th day, ask the users to opt in if they want to continue using the tool.
   • To control costs, we decided it was worth asking users to explicitly opt in instead of opting them in by default. This decision was based on the stats which I discuss in the results section.

Early Results

This data is based on 1 month of usage

• We had ~180 users who had made a commit in the past 60 days which set the top end of the budget estimate.
• We had ~140 users who signed up for the trial.
• Around ~100 of those users downloaded the Copilot plugin for their IDE within the first 30 days.
• We had 36 out the 100 users respond to the internal survey. Here are the results for some of the questions
  • When using Copilot I am more productive
    • 20% strongly agree
    • 53% agree
    • 22% neutral
    • 5% disagree
    • 0% strongly disagree
  • Roughly, how much time have you saved using Copilot?
    • No Savings: 5%
    • 0-1 hour: 45%
    • 1-4 hours: 33%
    • 4-8 hours: 11%
    • 8+ hours: 6%
• Around 60 people opted in to continue using the tool after the trial period.

Areas for improvement / Learnings

• One distinction I observed between my experience using Copilot in an experimental context and using with it across various work repositories was its superior performance when employed to build solutions involving external APIs or services, as opposed to building a solution from scratch. To illustrate this point further with a theoretical example, my feeling is that Copilot will excel when integrating AWS S3 service APIs into my code, compared to scenarios where I, as a developer might attempt to create S3 from the ground up, say, while working on the AWS S3 team. I do remain optimistic that this performance discrepancy will improve over time. The key takeaway from this experience is that if you are evaluating this tool, a diverse set of teams will need to use it for you to get a feel for its usefulness across the company. We could have set better budgetary expectations if I had realized this earlier.
• The Copilot stats were not available in the enterprise dashboard at the time of our trial. This meant that I had to reach out to GitHub to get the stats. As previously mentioned, the stats were only available for Visual Studio Code users, but we are a major JetBrains shop (I believe the stats for JetBrains will become available in Sep/Oct 2023). I would have liked to have a dump of the stats so that I can figure stuff out for myself instead of relying on the GitHub account manager.
• Copilot X was still in Beta, and it came with a different license. The tool seems very useful, but I decided not to derail the trial of Copilot by adding Copilot X to the mix and adding another license and tool that will need to be scrutinized by legal and security.
• The discount offered was ~2.5% for prepaying 25,000$. Considering that CDs are paying ~5% interest, prepaying was a losing proposition. I would have liked to see a better discount.
• The cost is 19$ per user per month. My personal opinion is that this seems a little steep for just Copilot. The pricing makes more sense once they GA more of their roadmap like X, Security integrations etc.

Other resources

• Getting Started with Copilot, User Guide
• Copilot Legal and Security FAQ
• Talk to your GitHub account manager about webinars. For example, they had one on legal concerns
• Black Duck Snippet Matching and Generative AI Models Recording
• Your GitHub account manager is there to increase your usage of the tool. They are a great resource to get your questions answered and we had an excellent one.

Disclaimer

• I am not a lawyer, this is not legal advice, please consult your legal and security team before using these tools