Read next

11 comments

• Melissa P

  August 22, 2023 9:55 am

  0

  I tried Copilot a few months ago and was not impressed. The AI drifts too much, I said “don’t use types, use ‘var'”. And the AI changed. Then I said refactor this and that. And boom, types were back. I said “make all local variable names lowercased pascal”. And the AI did that. Then I asked for another refactor and boom the new variables where uppercased pascal. 1 step forward, 2 steps back. When ever I instruct something, the AI breaks something else.

  Simple examples work and feel like magic, but the moment it gets a bit more complex, it can get frustrating very quickly. Also the AI is over-trained with the same primitive examples of Northwind database, a pizza delivery system with a shopping cart, and a people management system with a person class with name, first name, age etc. It “spills” into everything that has nothing to do with any of the 3.

  Also, the text to write/instruct is way too long as are the answers, to use it in a productive way. “root1 is Nan, why?” — “parameter of Math.Sqrt() was negative” — “oh, fix that with a catch-if” — “done”. Easy, simple, quick. I use ChatGPT4 a lot and I think the text I write the most is “short answers, please”. I know you all mean well but you have to be careful not to explain to developers with a masters degree the difference between a mouse and a keyboard.

  Log in to Vote or Reply

  • August 22, 2023 10:04 am

    1

    Hi Melissa

    Thanks for taking the time to try Copilot Chat, and for your feedback.

    We are just getting started with Copilot Chat, and will be working on improvements to make the system more useful over time. Whilst it does have limitations, we believe it can be useful for many cases. Thanks for giving us a sense of your priorities with this post; I’m hearing you’d appreciate the following features:

    1. If you have (or express) a convention in your queries, you’d like a way to make that “stick” for future queries on that codebase. E.G if you prefer var to types, you might prefer to have it honor your stated preference in your questions, or perhaps even from your .editorconfig file if supported?

    2. You would appreciate being able to ground Copilot’s responses in more complex cases (perhaps even your own company/org’s repos?)

    3. You’d prefer a more terse form of answers in explanations. You can already control this to some degree by asking for it in your question (give me a terse description of ….) but it sounds like you’d like a way to control it globally as a personalization of the responses?

    I’m happy to carry on this conversation on email too – my address is mwthomas at microsoft dot com,

    Thanks
    Mark

    Log in to Vote or Reply

    • Melissa P

      August 22, 2023 1:48 pm

      1

      I nowadays mostly use GPT4, but to my knowledge Copilot is using that in the background. With GPT I usually start a session and give instructions, which I now can pre-define. Short answers, reduced comments and compact/condensed code. And I tell GPT if I currently work using C#, C++ or HLSL. It works quite well but still from time to time I have to remind GPT to get back to my settings.

      I think for me personally, the most important thing is consistency above all. Every company defines its own rules of code writing that everyone has to stick to, so GPT needs to follow these rules strictly as well. As the conversation is verbal, it should be verbal was well to give these instructions, but I dont think that it really matters. It should be a one-time thing to say it has to be this or that way and GPT needs to stick to it.

      And if I may make some comments about the last video with the quadratic roots: The answer is so long, it doesn’t even fit the screen. I would have to scroll up and down to read everything. And most of the text is written like conjecture and “it may be wrong” and “please don’t sue if it’s wrong”. I understand the reason why now everything needs a mile long disclaimer but I just don’t want to read them again and again. And then finally the suggestion “can you explain what a quadratic equation is?”. It makes me feel bad, as if I don’t know what I’m doing and just stumble upon some code I don’t understand. That’s not what I want. I want GPT to consider me as someone who knows all about what’s going on and just have some brain freeze and don’t see what the problem is and GPT says “you forgot about the negative determinant” and I say “omg, of course, how did I miss that obvious mistake” because that’s what it is, or should be. The level of professionalism is not equal, if that makes sense. I want Copilot to support me, not to train me. I have always the option to ask further questions. To get more detail. And I want to have this more reflected in the AI responses. “can you tell me about all special cases for quadratic equations”…. “can you write the root formula for quad equations” .. “can you verify the equation i use is correct” “can you give some numeric examples to test my code”…. these are so much better question templates/links than “what is math”, “what is an equation” … I dont know if I make sense here, I just want to be “treated” by the AI as a professional and not a first time student.

      And don’t get me wrong. We all started from scratch some day ago and had to get a Hello World running. But most users will have moved on. And just don’t see what’s wrong with some current code or think about the issue at hand. Why the code isn’t working.

      Today I asked GPT “what windows API tells the system to not go to sleep?” — I used that function a hundred times, it has a weird name, I never remember it. “The Windows API function SetThreadExecutionState is used to inform the system not to enter sleep mode or screen saver mode.” was the reply. Short, precise, no fluff. It saved so much time, it is so helpful. And I think that’s the case for most of us. We all know exactly what we want to do, we just lack that little detail, the name, the parameter.

      Last week I needed to check if a line intersects a rectangle, I’ve done that a hundred times, it needs like 8 checks but it’s hard to remember all of them. I was lazy and didn’t wanna bother with it, so I asked GPT to write the code for me. I didn’t need a mile long explanation what it does, I know exactly what it does or how it works; I was just lazy and why shouldn’t I, I’ve done it many times but didn’t want to waste time on searching or copying my old code.

      It’s a long text, but these are the things I want from Copilot. I wrote GPT but it’s the same. Support not teaching. When something is unclear I can always ask for more information.

      Log in to Vote or Reply

      • August 22, 2023 4:20 pm

        0

        Thanks so much for the extra detail. It helps us understand where to invest next.

        Log in to Vote or Reply

• MgSam

  August 22, 2023 10:22 am

  0

  I’ve been using the preview Copilot / Copilot Chat since Build and unfortunately it has been a massive disappointment. It doesn’t work anything like it does in Microsoft’s demos.

  – The in-context Copilot features just don’t work at all. I almost never get code suggestions and it doesn’t attempt to suggest code based on comments. Even when it does offer suggestions, its simple stuff like a semicolon. I’ve never had it offer to complete an entire method as is shown in the demos.
  – Copilot Chat is frankly awful. It is incapable of analyzing even the most simple of methods in my experience, and rather than tell you that, it just completely lies and makes up details about the method that are not true. I asked it to tell me the difference between two similar methods and its explanation was literally 100% BS that had nothing to do with what the methods actually did.
  – 70% of the code that it generates is completely wrong and useless. I’m assuming what is happening is that in my large solution, VS is passing too much context and the GPT is choking and dying on it.
  – After 5-6 prompts with Copilot Chat is goes berserk, stops answering prompts in sentences and starts responding with largely irrelevant nonsense.

  I’m guessing that, as usual, Microsoft is only testing with toy “todo list” apps rather than anything resembling a real-world solution with many projects. I’ve found it to be substantially inferior to just asking ChatGPT-4 targeted questions.

  Log in to Vote or Reply

  • August 22, 2023 10:30 am

    1

    Hi MgSam

    We have been updating Visual Studio Copilot and have made a number of changes since Build in the preview bits. Based on testing and feedback we have a number of improvements planned in the area of context management too. The new inline chat feature makes context easier to control too – I encourage you to try it.

    We are still in the preview phase, and would very much welcome your bug reports from Visual Studio via the “report a problem” feature, including the Copilot Chat logs from your Output Window’s “Copilot Chat” pane – if you have cases like the ones mentioned above please raise feedback tickets and we’ll dig in to understand and fix.

    Thanks
    Mark

    Log in to Vote or Reply

• George Belletty

  August 22, 2023 3:11 pm

  2

  I’m liking Copilot a lot, I’ve had it in Visual Studio for about 5 weeks now and I didn’t really use at the beginning. I started using it by replacing my searches on google and stack overflow for syntax or finding the best way to do something.

  The suggested code has saved me probably thousands of key taps a day! You know when you start writing a class and like intellisense, it suggests properties given the name of the class.

  My next project is going to have a lot of handler classes… I wonder if I could write a couple of them, keep the files open in VS and then ask Copilot to create more for the crud operations on the other entities. That could be interesting.

  Looking forward to see how Copilot evolves over time.

  Log in to Vote or Reply

  • August 22, 2023 4:19 pm

    1

    I’m glad to hear Copilot is helping you so far! Would love to hear how your experiment goes; we are looking at further improvements to context going forward and your case is an interesting one. Please do share – you can drop me a line anytime mwthomas at Microsoft dot com or raise suggestions and feedback via “report a problem” in Visual Studio.

    Let us know what you think of the new inline refinement experience?

    Regards
    Mark

    Log in to Vote or Reply

• The Werewolf

  August 22, 2023 5:05 pm

  0

  I assume this can be disabled because God knows there’s nothing I want more than a half-baked back seat driver looking over my shoulder and making questionable suggestions on how to code better and triggering my ADHD.

  Guys, maybe fix the problems VS actually has before gluing more stuff literally NO ONE asked for into it?

  But I DO have to say THANK YOU for putting a feature to move the menu bar out of the title bar. Been begging for that for so long…

  Log in to Vote or Reply

  • August 22, 2023 5:59 pm

    1

    Hey there!
    This feature is currently in preview and requires an explicit sign-up step to get involved, so you certainly do not have to use it.

    While I do not imagine that all developers will find this equally helpful, I am seeing a lot of early evidence that some developers are already relying on conversational assistance while coding. We just want to make sure we have the right tools available for the many kinds of developers we encounter.

    The goal is to fix bugs and respond to community requests while also delivering new features for the broadest variety of Visual Studio developers.

    Thanks for the feedback on the menu bar
    Mark D

    Log in to Vote or Reply

• Steve Tabler

  August 23, 2023 8:18 am

  0

  Is this available for any version of Visual Studio? Becuase I need to generate .NET 3.5 code all the time, and my most comfortable environment is Visual Studio 2008 for this. I use ChatGPT V2, occasionally for this, more for a sounding board to get ideas when I’m kinda stuck on how to approach something, and ChatGPT keeps forgetting that I specified .NET 3.5 coding, and outputs uncompileable solutions that I have to figure out. This would be less of an issue if someone wasn’t making it a point of erasing all the old VS2008 and .NET 3.5 help-pages and content from the internet. And I don’t have a budget for using a paid AI-service of any kind.

  Log in to Vote or Reply