Copilot Enterprise - Private Beta Feedback

[iamcjwilliams]

Copilot Enterprise - Private Beta Feedback

• Who: GHEC Customers currently participating in the private beta for Copilot Enterprise.
• What: We want to hear from you! Provide feedback on the Copilot Enterprise Private Beta in this Discussion

---

The purpose of this discussion is to collect feedback from our GHEC Customers that are currently participating in the private beta for Copilot Enterprise.

• To access the private beta, please sign up for the waitlist.
  • The waitlist is only open to Copilot GHEC customers.
• Interested enterprise customers can reach out to your GitHub account manager or contact our sales team for enablement.

This thread is just for product feedback.

We are responding to Questions & Answers on the private beta in another discussion: Copilot Enterprise Q & A

[rc-ian-drennan]

We are just exploring Github Copilot Enterprise and one of the first things we noticed is that it's difficult (well not convenient) to use PR summaries in our system due to the fact that we make extensive use of PR templates.

It would be great of PR templates supported "AI Markdown" that allowed us to mix pre-written text along with copilot generated text.

For example something like

Some precanned text, autogenerated text from our internal tooling. HTML tables etc

copilot:summary

Detailed PR analysis

copilot:walkthrough

[arfon]

Thanks for the feedback @rc-ian-drennan. Tagging the Copilot for Pull Requests PM (@abhinavvadrevu) here for visibility.

[rc-ian-drennan]

You're welcome - we're excited to dig in and get the most out of this!

[abhinavvadrevu]

Hi @rc-ian-drennan! Thanks for sharing this feedback! I'd love to learn a little more about this usecase. Could you share exactly what precanned text you would want to insert? Would it be some in the form of some questions that copilot should fill out answers for?

[rc-ian-drennan]

Hi @abhinavvadrevu,

Happy to share. We use templates (like the sample attached) where we would want "different" sort of sections generated by CoPilot. We have other automation that triggers to add JIRA links, etc which is why having a CoPilot markdown symbol could be really helpful.

Example template
We generate on PR creation this HTML block

Jira Ticket Title

Summary	Text from the JIRA ticket
Type	Type of JIRA Ticket
Points	Points from the JIRA ticket
Labels	Labels from the JIRA ticket-

Summary

We would want a high level CoPilot generated summary here

Description

We would want a detailed CoPilot section here (much like it generates today)

Type of Change

<..Snipping the rest of a bunch of boiler templating...> Nothing to see here this is company specific stuff

Testing

We would want a summary of tests and testing changes here

[rc-ian-drennan]

So if we were to write this markdown in our .github section we would want to write it something like this (pseudocoding for example)

Jira Ticket Title

Summary	Text from the JIRA ticket
Type	Type of JIRA Ticket
Points	Points from the JIRA ticket
Labels	Labels from the JIRA ticket-

Summary

copilot-summary

Description

copilot-description

Type of Change

<..Snipping the rest of a bunch of boiler templating...> Nothing to see here this is company specific stuff

Testing

copilot-test-summary

[alshain]

In IntelliJ, it would be nice if postfix completions didn't compete with GH Copilot suggestions concurrently. Depending on whether completions from GH have already loaded or not, it will either expand to the postfix completion or to the GH Copilot suggestion.

[cafedomingo]

Minor issues, but when using the dark themes for github, the syntax highlithing in the copilot chat is really hard to read with some of the text colors being nearly identical to the background color.

[cafedomingo]

"Dark dimmed" is unusable.

[alwell-kevin]

For the DocSet beta, I would expect the interface to live within Org/Copilot Features, or some similar place.

I'd also like to be able to delegate docset creation to non-org admin users. Possibly through a custom role.

[arfon]

/ cc @timrogers

[timrogers]

Thanks for the feedback @alwell-kevin 🙇🏻 This is definitely something we'll think about as we iterate on Copilot for Docs.

[JannesPeters]

Copilot asks me to provide a link, then says it can't read links :D

[arfon]

Ah yes, we should teach it how to do that!

[JannesPeters]

Do we also have plans for Copilot to be aware of the page I am looking at? If I ask questions like "like summarize this" or "summarize this issue" it currently doesn't seem to be aware of what I am looking at.

[bmcmanus-apex]

I would love if it had access to the page, specifically PRs as Copilot could be leveraged to improve the PR review process.

[JessRudder]

+1 to this. I haven't used the "Ask Copilot" feature at all until today. I was on a PR, opened it up and asked it "What is this PR trying to do?" It's response was that it didn't know which PR I was referring to and I needed to give it a number or a link. I gave it the number and it told me it couldn't find it and asked me to verify that it was correct. I gave it a link and it said it couldn't find that either.

I understand that it might not be able to do what I was asking, but, it would have been a better experience if it stated that clearly. The feedback it gave me made me think it could do what I was asking as long as I gave it the proper info.

[michaelhoffmanalight]

In order to sign my organization up for the beta, they are looking to understand the policies that are either being continued or changing for Enterprise. For example, they require re-assurance that custom models could not share / link IP. It would be helpful to have another "About" page for Enterprise, similar to the ones for individual and business (https://docs.github.com/en/copilot/overview-of-github-copilot/about-github-copilot-business). This way I can point my organization at the page and assure they can directly review the privacy and security measures related to Enterprise.

[arfon]

This is great feedback, thanks @michaelhoffmanalight. / cc @iamcjwilliams and @2percentsilk here.

[shehbazkhanani]

Any way to use free Github Copilot

[nfurnosanofi]

Feedback on Copilot Docsets

Our team built a repo that contained a readme.md. This readme.md contained the details on our shared ARC runner tag to be used in github actions, along with our standard environment variables and proxy configuration to be used in a standard github action developed by our teams.

I put this lone repo into a docset and clicked index.

Once indexed, I used a browser based copilot chat window in the context of this docset, and asked it to produce me a basic github action. It did not use the information in the readme to product the action.

I asked again, more specifically, what is our companies standard runner tag for a github action? It was unable to answer and informed me that it does not have access to data in our repository.

We then tested this same thing using the repo itself as the context (not the docset). and got the same responses about being unable to see the data inside the repository.

I'm curious if perhaps we're using the product incorrectly? What steps might we take to troubleshoot our docset implementation?

Thanks,
Nick Furno
Sanofi

[imjohnbo]

Thanks for the question, @nfurnosanofi. Let us look into it and reply to you shortly.

[timrogers]

@nfurnosanofi Thanks again for taking the time to share your feedback, and sorry for the delay in getting back to you 😓

From what you've said, it sounds like information about your standard runner tag is in the README.md, so Copilot Chat should be able to read that. I'm wondering if your repository isn't or wasn't indexed properly, as that would explain the issue.

Could you try this again now, and let me know if you're still having issues?

[Preskton]

Howdy, folks - super excited for copilot and have already had some great success with minimal effort. One particular use case I'm interested in is helping people connect to other people, and to that effect, we have repos that have markdown files that are essentially subject-to-people mappings, e.g. if you need to know more about the Torment Nexus, talk to @Preskton.

When I ask co-pilot these types of questions, like:

which architect should i talk to if i need to know more about the torment nexus?

copilot seems to short-circuit and provides answers like:

I'm sorry, but as an AI, I don't have access to personal information about individuals or their roles at a company.

I get that this is likely a function to protect privacy or protect against AI-based libel/legal issues. How could we enable this type of very valid use case, though?

[arfon]

This is very interesting feedback, thanks @Preskton. Did you create a docset with this Markdown content?

/ cc @timrogers for visibility

[gr2m]

Sorry for the noise, just wanted to say I asked a non-trivial question about the probot/probot repository and Copilot aced the answer, very impressed. Great work team!

[arfon]

Great, thanks! Curious what the question was (if you recall/can share)?

[bmcmanus-apex]

First off, I really love the PR summary functionality. I've been sending git diffs to ChatGPT for far too long to get this functionality. It would be great if there was an option using the gh CLI that would allow it to fill the description with a Copilot summary. The only thing I do not like so far about Copilot summary functionality is all the guessing language as to why a change was made; things like "which suggests", "likely indicating", "as some reason (that has been 100% wrong so far)". Remove the speculative information in the summaries and ya'll nailed it.

[arfon]

Thanks for the feedback @bmcmanus-apex / cc @abhinavvadrevu

[HansFalkenberg-Visma]

Is this the right place to report bugs also?

I have indexed 3 repositories on github.com and the chat is acting inconsistent about actually using those indexes, often refusing use them.

For the first two repositories, the indexing started straight away. For the third repository it asked me what repository I wanted to use, although I was on the root page of the repository when I opened the chat, just the same as I had been with the other two. I did get to select the third repository though. After about 10 minutes all three repositories started reporting success:

Indexed for improved understanding and accuracy."

But when asking questions about the first two repositories, eg. "Give me an overview of this repository." I get general answers like

I'm sorry, but I can't provide specific details about the code in the ORG/REPO1 repository as I don't have direct access to the codebase.

When I asked how to use feature toggles I also got a general answer, but when I followed that up with "In this repository" I got an answer based on repository contents. So it will sometimes use the repository contents, but most times not.

For the third repository, I got what might be a variant on the answer above, but one that sounds a little worse:

I'm sorry, but I couldn't find any further details about the repository 'ORG/REPO3'. The repository might be private, not exist, or there might be some other issue preventing me from accessing the data.

[arfon]

Thanks for the feedback @HansFalkenberg-Visma. Can you say a little more about how you were interacting with the various repositories when asking questions like this? For example did you have the side-panel (assistive mode) of Copilot open on the repository you were discussing?

Screenshots (if appropriate to share here) are always welcome!

[HansFalkenberg-Visma]

I opened the root URLs for 3 repositories (https://github.com/ORG/REPO) in different tabs and then clicked the Copilot icon and started indexing of each of them. The REPO names were ...-infrastructure, ...-backend and ...-frontend. It was the ...-frontend which was the REPO3 above which it initially asked me to select what repository I wanted to index, although that should have been clear from the URL.

I re-asked some questions in the same way now.

First off, it is very confused about which repository I am looking at...

It happens to be an SPA, but I find myself far from trusting that CoPilot knows this instead of just stating random stuff based on the fact of the repository it's describing being named "...-frontend":

Especially given the apparent incompetence illustrated by the next answer:

I changed the URL and this time it got the repository context right:

But it won't use the actual code to describe anything. Also, it changed the name from kebab-case to Kebab-Case in the title. A programming assistant that can't even get casing right is easy to mistrust:

As encouraged to I asked about a specific aspect, but again the answer is worthless:

[arfon]

Thanks for the feedback @HansFalkenberg-Visma! Right now, the types of questions we best support are fairly targeted ones. So rather than things like "Give me an overview of this repository." or "Is the infrastructure in this projected target for cloud native", we support questions like "Where is the cloud formation script to deploy this application" or "where is the authentication implemented in the API".

We are doing some work to explore a more general 'high level' summarization capability for Copilot, but right now, the most supported scenarios are those to explore code.

Hope this makes sense (and keep the feedback coming ✨)!

[dluc]

I love the chat to "talk to the repo", please make it easier to discover and to trigger.

For instance, it would be great having a badge in READMEs to open the Copilot UI

[arfon]

Oh, that's a really cool idea. / cc @skylar-anderson

[kevingduck]

This is great! I asked for tips on optimizing a long python script. Seeing some tips and then having these other users' specific projects recommended like this is super smart.

[darjoo]

Can't seem to get a summarization of PRs.
The PRs are definitely open as I was viewing them when I asked the question. If I just provide the pr number, it says it can't access or summarize specific PR..

[abhinavvadrevu]

Hi there! PR summaries are available as part of the text formatting toolbar in the "Description" box when a PR is being created. We don't currently have the ability to summarize a PR from chat. Hope this helps!

[darjoo]

Ah okay. Thanks!

[corycalahan]

When opening Issues in the new Issue Create experience, the Copilot chat window interferes and hides buttons I wish to interact with.

Screen.Recording.2024-02-15.at.1.55.14.PM.mov

[arfon]

That's not good! Thanks for spotting that UI issue @corycalahan.

[not-an-aardvark]

An organization that I'm a member of recently gave me access to Copilot due to the fact that I'm in the organization. As a result(?), an "Ask Copilot" widget is appearing at the bottom of many pages on github.com:

Personally, I'm not interested in using Copilot now or in the foreseeable future, and I'd like to permanently hide this widget. I've tried pressing Escape or the "x" button, and the widget goes away temporarily, but comes back a few days later. I've looked in https://github.com/settings/copilot for a way to disable it, but didn't see an option.

The "give feedback" link on the widget pointed me to this feedback, so I'm hoping this is the right place to request an option to disable the "Copilot ad on every page" feature.

[arfon]

The "give feedback" link on the widget pointed me to this feedback, so I'm hoping this is the right place to request an option to disable the "Copilot ad on every page" feature.

Thanks for the feedback @not-an-aardvark. We are investigating a way for users to be able to more reliably disable this UI element. Sorry for any frustration caused here!

[ericwbailey]

+1. I'd expect a user preference for this and have resorted to blocking it with a user style.