I really doubt it would be faster to write code manually, even with the state of AI tools today. Even with very sophisticated keyboard macros and traditional autocompletion, someone using GPT would outperform anyone who doesn't. Think of the amount of boilerplate and tests you write, and tedious API documentation lookups you do daily; that all goes away with GPT. The amount of work to double check whether the generated code is valid, and fix it, is negligible compared to the alternative of writing it all manually.
Of course, I'm saying this without actually having used it for programming, so I might be way off base, but the feedback from coworkers who rely on even the now basic GitHub Copilot is that it greatly improves their productivity. I'm envious, of course, but I'm not willing to sacrifice my privacy for that.
I agree with this statement a lot. Using Copilot saves you a lot of tedium if you are comfortable with the language already. If you are new to the language, then it might trip you up a bit (at least in its current incarnation).
Here is an example where it helps. I tried to initiate a connection to a Mongodb server using Python. While i have used many databases before, I have never used Python and MongoDB together. So, i knew i would have to have some kind of MongoDB library, a connection Factory and a connection string. I could have googled all of these things.
I did the following in VS Code using CoPilot.
def get_db():
"""Initialise a MongoDB connection to a local database"""
It then automatically filled in the rest.
db = getattr(g, '_database', None)
if db is None:
db = g._database =
MongoClient('mongodb://localhost:27017/')
return db
Notice above, that it knew i was using a flask environment and added the line getattr.
Why this is a productivity boost is that i did not have to alt-tab to a browser, search for "pythong mongodb tutorial example" and then type it out. I was able to do the whole thing from VS Code and since i use vsvim, i could do this without taking my fingers off of the keyboard.
This is the next jump since autocompletion. I like it.
And you will have no idea whether the solution it presents to you is idiomatic or recommended or contains some common critical flaw or is hopelessly outdated. How can you find out? Back to alt-tabbing to the browser.
Sure it may take a bit more time to get going, but then you'll get it right the first time and learn something along the way. Your copilot example is just another iteration of copy-and-paste some random snippet from StackOverflow in the hope that it will work, but without having seen its context, like from when is the post and what comments, good or bad, did it get.
I'd actually be pretty afraid of a codebase that is created like that.
> You have no idea if the alternative code you would have written would have been idiomatic or had some critical flaw.
But I have a feeling for both, which is one of the key components of the skill in our trade.
For idionmatic code, I know the degree to which I'm following how things "should" be done or are "usually" done in a given language. If I'm uncertain, I know that. GPT won't tell me this. Worse, it will confidently claim things, possibly even if presented with evidence to the contrary.
For critical flaws, I know the "dark corners" of the code. Cases which were not obvious to handle or required some trick, etc. I'll test those specifically. With GPTs code, I have no idea what the critical cases are. I can read the code and can try to guess. But it's like outsourcing writing tests to a QA department. Never donna be as effective as the original author of the code. And if I can't trust GPT to write correct code, I can't trust it to write a good test for the code. So, neither the original author of the code (GPT) nor somebody external (me) will be able to test the result properly
I mean... I certainly know which languages I can write idiomatic code in and which I cannot.
I can't know that my code will be free of critical flaws, but I do understand the common sources of flaws and techniques to avoid them, and I'm quite confident I can build small features like this that simply aren't vulnerable to SQL injection, on the first try and without requiring fuzzers or code review: https://infosec.exchange/@malwaretech/110029899620668108
I'm confident enough in most languages I write in to recognize correct code. But I am not usually so familiar that I can conjure the exact syntax for many specialized things I need. Copilot is just a much quicker way to get what I need without looking it up.
You don’t have to accept the suggestions as-is. It’s just code, you can edit it as much as you like.
Getting a good idiomatic starting point is a great boost.
Watch the demos where they provide GPT-4 with an API for performing search queries and calculations. These tool integrations are the next step and they will include specialized ones for using language and library docs. They could also be given access to your favourite books on code style or have access to a linter that they could use to cleanup and format the code before presenting it. The model is capable of using these tools itself when it is set up with the right initial prompt. Even now Copilot is pretty good at copying your code style if there is enough code in the repo to start with.
It is. I can see that it was written in 2003 and discard it. GPT won't tell me if its answer is based on an ancient lib version.
Essentially, GPT is that rando's webpage but with the metadata stripped away that allowed me to make judgement calls about its trustworthyness. No author, no time, no style to see if somebody is trolling.
> Think of the amount of boilerplate and tests you write, and tedious API documentation lookups you do daily; that all goes away with GPT.
At work we have really worked hard to minimise boilerplate and manually-written/repetitive tests, so I don't write much of that. Getting GPT to write it would certainly be worse: we would still have the deadweight of boilerplate/repetition even if we didn't have to write it, and some of it would be incorrect. Maybe this varies a lot by company — if you're often writing a lot of repetitive code, and for whatever reason you can't fix the deeper issues, then something like GPT/Copilot could be a godsend.
About documentation lookups, I don't know if this varies by language, but I've had very little luck with using GPT for this. For the languages I use regularly, I can find anything I need in the documentation very rapidly. When I've tried to use GPT to answer the same questions, it occasionally gives completely wrong answers (wasting my time if I believe it), and almost always misses out some subtlety that turned out to be important. It just doesn't seem to be very good for this purpose yet.
> At work we have really worked hard to minimise boilerplate and manually-written/repetitive tests, so I don't write much of that.
There's boilerplate in any codebase, even if you make an effort to minimize it. There are always patterns, repeated code structure, CI and build tool configuration, etc.
If nothing else, just being able to say "write a test for this function", which covers all code paths, mocking, fuzzing, etc., would be a huge timesaver, even if you have to end up fixing the code manually. From what I've seen, this is already possible with current tools; imagine how the accuracy will improve with future generations. Today it's not much different from reviewing code from a coworker, but soon you'll just be able to accept the changes with a quick overview, or right away.
This may be highly dependent on problem domain or programming language (see the other article about GPT tending to hallucinate any time it is given problems that don't exist in its training set). My experience has mostly been that the output (including simple stuff like "test this function", though we generally avoid unit tests due to low benefit and high cost) is consistently so flawed that the time to fix it approaches the time to write it.
> I really doubt it would be faster to write code manually,
Not faster or slower but at what quality?
At least every time I've tried to ask GPT 3 & 4 to write anything it's always missing things or not even close to the optimal way that I have to look up the docs and fill in the gaps, which often takes just as long as starting from scratch.
> now basic GitHub Copilot is that it greatly improves their productivity
Perhaps it depends on what you're working on. If it's quick iterations that isn't that concerned about what code goes in and whether it's maintainable then sure.
It's impressive but for now still has lots of gaps. However it is over confident and often misleading.
I've had similar experiences when testing it out with Rust, Java and Go. Once I got beyond basic stuff, very little of the output was of a quality that I would consider remotely acceptable, and the work to bring it up to standard was basically equivalent to just writing the code in the first place (which, come on, typing is not even the time-consuming part of engineering).
It makes sense that it wouldn't be very good at Go or Rust since both are rather rare languages in the open source world that copilot is trained on. When I did my first go project a few years ago, I had the hardest time finding even basic examples like how to parse a json string. Rust is even newer and less used. But java is the 3rd most popular language for GitHub projects so I would think it would do better with Java.
There are over 50,000 and over 40,000 repositories just on GitHub that contain Go and Rust code, respectively. [1] Among them, some truly massive projects like Kubernetes for Go or Servo for Rust. I will freely admit your argument for new and/or obscure languages like Hare, but Go and Rust are not "rare" under any reasonable definition of the word.
The failure mode seemed similar in all three languages. If you were doing toy things, or writing boilerplate stuff, it did perfectly fine. If you were writing something that wasn't a slightly-modified copy of some code that already exists out there, it fell apart. I don't think the issue is the language in this case — Go and Rust are common enough, and it rarely had trouble with the syntax — I think it's that the model doesn't go very "deep", so it's able to reproduce common patterns with minor variations but is unable to conceptualise.
Java is ultimately worse. It's old. It's gone through A LOT of change over the years. How does it even tell what's good and bad? Java also has lots of convention over configuration and "magic" in most frameworks, which it doesn't exactly understand.
When I tried it I'd often have to go back to it and keep telling it to use a different way of doing things because the world moved on. By that time there wasn't much point.
I see praises by those that have never coded in a language. They try ChatGPT, see it produce "ok" output and call it a day. If that's where we're going the web will be even more bloated than Electron and everything will be 10x worse. It's like low code but even lower (in quality).
If you eliminated 100% of my code typing time with perfect effectiveness I think that'd make me maybe 10% or 20% more productive? Turning ideas of what the code should be doing into code just isn't a bottleneck for me in the first place. Are there people who just add net 1000 lines of code to whatever they're working on every single day or something?
I keep seeing this point made, but AI tools don't save you just typing time. They save you time you would previously use to lookup documentation, search the web and Stack Overflow answers. They save you time it takes to navigate and understand a codebase, write boilerplate code and tests, propose implementation suggestions, etc.
Dismissing them on the basis that they just save you typing time is not seeing their full potential.
there are people who code the whole thing in their head and have perfect recall of all language/api docs/references/syntax/features as that is how their brain works. ie they don't even type code until this step has occurred for them.
I think that is a small percentage of the dev community, so for me and people that don't operate like that, these tools are a game changer as you point out. I don't take what ChatGPT says at face value, I've got 15years of experience I'm weighting results against as well...
chatgpt's version of the above:
Coding entirely in one's head is rare. Most developers need external resources, making development tools invaluable. While ChatGPT's input is valuable, it should be balanced against personal experience and expertise.
Navigating and understanding a codebase is the only one of those which would excite me, but it's also something I've never even someone propose using ChatGPT for. Do you have an example of what that would look like?
My experience was always that most of my time isn’t deducted to writing code. Maybe 10%, the rest us thinking about how the code I write will fit into the existing architecture or accommodate future features.
In very near future your IDE will send the whole codebase as context to LLMs. Then instead of thinking up all possibilities you can just ask. LLM will suggest multiple alternatives and you can select the best when and ask it to implement it.
I said that the context window improved. I mean that it is larger. GPT-3.5 is 4k tokens, GPT-4 is 8k tokens (standard) or 32k tokens (only API access atm). This is the number of tokens that GPT-X can take into account when producing a response.
Specifically, I was using this to support the statement "In very near future your IDE will send the whole codebase as context to LLMs." I'm not talking about loops or accuracy.
It's true, but there is no indication that GPT can explain larger concepts for you, and negative indication it will be able to do it accurately.
It can't even explain small code to me unless it is something that it has been trained on. Often it gets even simple things wrong, either obviously, or worse, subtly wrong.
I agree that this is the part that needs more work, and is most uncertain. Increasing context windows seems like a fairly straightforward computational challenge (albeit potentially expensive). On the other hand, whether or not we can scale current models towards "true understanding" (or similar), is a total unknown atm.
I still think we will get useful things from scaling up current models though. I've already got a lot of value out of Copilot, for instance, and I'm looking forward to the next version based on GPT-4. Recently, I've been using the GPT-3 Copilot to write a lot of pandas/matplotlib code, which is fairly straightforward and repetitive, but as mainly a Java developer, I just don't have the APIs at my fingertips. Copilot helps a lot with this sort of thing.
GPT-4 has a model capable of using around 50 pages of written text worth of tokens (32,000) not sure exactly how many lines of code that translates to but it’s a lot. GPT-3 can use 4k so that’s a huge increase, the next version could be even larger and there are other ML techniques that allow for massive context lengths. Copilot already does a good job of refactoring code and knows enough about your code base to use your functions and methods. So what the other commenter said does not sound impossible to me.
It isn't anywhere near being able to diagnose anything more than off-by-one and other common errors.
Identifying hose problems will bring a LOT of value - but it isn't going to program and do general problem solving for you! It just has no signs of being able to do that.
Yesterday’s Steve Yegge post talks about it. You can provide text as context but you can also provide a dense representation in the form of text embeddings that capture the context. Today you can manually do it by something like LangChain but in future it will be part of our text editors.
Yes it will be able to give amazing feedback to us as devs and quickly identify common problems (that I still make all the time, even after developing for a decade!) which will bring a ton of value to programmers.
But, it will be a tool - it won't be something that will solve general problems for you. It won't make an average programmer a great programmer.
The moment that happens we can all forget about working and just do arts, space exploration, and acid orgies. But that “future” is somewhere between full self driving and thermonuclear reactor.
Of course, I'm saying this without actually having used it for programming, so I might be way off base, but the feedback from coworkers who rely on even the now basic GitHub Copilot is that it greatly improves their productivity. I'm envious, of course, but I'm not willing to sacrifice my privacy for that.