As a computer scientist and computer programmer with over 2 decades of experience in different software disciplines (Robotics, web dev, computer graphics, game development, system analytics), I decided to put chatGPT to the test. The technology that junior developers and computer science students fear might make them obsolete very soon.
I have been using chatGPT for the past 2 weeks. I played around a little bit with DALL.E 2, but most of the time was spent on chatGPT. I wanted to see for myself what the hype was all about. I got tired of hearing people talk about it, so I decided to see things for myself. I will go into some details concerning the problems I tried to get chatGPT to solve. I know a lot of developers who used chatGPT for some web dev stuff, both back-end and front-end. They got some really impressive results. I thought I’d task chatGPT in other non-web dev areas.
I am currently leading a team of engineers, building complex software used for post processing effects in movies and short films. We have a codebase of nearly half a million lines of code. We write a lot of Vulkan, OpenGL and Metal code, while using Qt framework for our GUI; Of course, it is all C++. I asked chatGPT some questions specific to our codebase but it kept on flopping badly and giving completely wrong and misleading answers, so I decided to try something else. I do a lot of testing in Unity engine and Unreal engine, because some of our clients produce movie scenes using these engines and then they apply finishing touches using our own software. I decided I will ask GPT some questions in Unity C#, Unreal C++ and anything in between.
I first asked chatGPT to write me a simple highlight shader in HLSL in Unity engine. It wrote some code that wasn’t working. I wasn’t getting any shader error but the highlight feature just wasn’t working. Before prompting chatGPT, what I expected was some implementation of the dot product in the shader code. Areas of the mesh with the highlight colour would be area where the normal vector of the faces are perpendicular to the view vector or almost 90 degrees away from the view vector. ChatGPT gave me a weird implementation that I was not expecting, using some formulas I have never seen before.
I then asked chatGPT to give me a working water shader in HLSL. I’m not even talking about volumetric simulation for water physics, just the plain-old water shader applied on a plane. This one failed miserably as it could not even compile. Some variables were not even declared and they were being used. I wondered why a “thinking” AI would do this. I told chatGPT about the errors I was facing and it could not solve them. I managed to fix all the errors myself. When I ran the shader code, I got something that did not look like or function like a water shader. After the back and forth with chatGPT, I then asked google for a water shader in Unity. I clicked on the first link google brought up and it linked me to someone’s GitHub repo. I copied and pasted the code from GitHub and it worked like magic. So I had better success with google than with chatGPT.
I then asked chat GPT to write a calculator that would work in Unity. When I say calculator, I mean a functioning GUI calculator that can do basic arithmetic without the whole sin and cosine functionalities. ChatGPT gave me a code that had no compile errors but when I ran the code, 6 + 4 was giving me 12. I immediately gave up and moved to the next prompt.
This one was the worst. I asked chatGPT to write me a working airplane controller. It produced a very simple flight controller that allowed me to move the airplane game-object left, right, front and back with the arrow keys on my keyboard. It was not based on any flight physics. I then asked GPT to produce a working flight controller based on some sort of realistic flight physics which would include forces such as gravity, thrust, drag and lift. I also asked chatGPT to take other variables such as coefficient of lift and angle of attack into consideration.
The code produced by chatGPT was laughable. It was so bad, I didn’t even know where to start. The formulas given for lift and drag were wrong and the entire flight physics was terrible. After I attached a “rigidbody” component to the airplane object and ran the code, the airplane immediately started moving backwards at the speed of the gravity value (9.81m/s), LOL. I decided to probe deeper.
I then asked GPT what fundamental forces act on an airplane. It listed out lift, drag, gravity and thrust and explained them correctly while giving the right formulas for all of them. So it gave the right formulas for calculating lift and drag forces.
So, If GPT knew all the formulas, why couldn’t it apply them in code? It does not look like chatGPT was doing any “thinking”.
I then asked chatGPT to give me a working solution for a small video program that would involve video encoding. The task was to procedurally generate a 3 seconds video that shows a red screen for one second, a green screen for one second and a blue screen for one second using the H.264 encoding technique. I also asked GPT not to use the ffmpeg library. The red, green and blue screen were to be generated procedurally.
It was generating some impressive code at first. It then started writing the “RIFF” header and other encoding standards for H.264. It then proceeded to write some bytes and then suddenly got stuck. I waited for an hour, then prompted GPT to generate the same answer. It got stuck again. I then wrote “are you not able to solve this problem?” It then provided a solution using the FFMPEG (which I initially asked it not to use). Maybe it was copying code from an incomplete repository. Who knows ?
Like I said before, chatGPT does not think. The GPT technology is about statistical wizardry to create a good work from the combination of other people’s work, from data it knows about. It does not provide solution to a problem that has not been solved over and over. This is the reason it is able to solve lots of basic web dev problems and coding interview problems easily. The source codes to these problems are all over Github and stack overflow.
Other problems I tried to get GPT to solve include, volumetric shader for clouds in HLSL, an inverse kinematics system for four-legged robots, compression algorithms for tessellation maps, multi-agent systems for virtual swamp robots searching an air-space and moving in coordination. It failed all of them woefully. I wasn’t expecting much. After all, it could not even provide a working solution for a GUI calculator in Unity C#.
CHATGPT does not think
Like I have said over and over. It does not know what it is doing. The tech is impressive from afar, get closer and the facade starts to fall apart. A lot of its understanding of context is very rudimentary and it’s ability to apply it’s vast knowledge on a practical level is minimal.
chatGPT will not get much better than this
Just like many other things, growth does not last forever. A peak is always reached. When you need to feed an AI system with tens of thousands of images of cats just so that the system can finally “understand” what the photo of a cat really looks like, such system is not really intelligent. chatGPT looks like it has almost got to its peak. With tens to hundreds of millions of dollars, years of research and an entire universe of data poured into it, including source code stolen from millions of developers world wide, it still could not write a simple highlight shader in Unity or solve any of the tasks I gave to it. These problems can easily be solved by any programmer with a decent game dev background.
GPT is overhyped for profit
The tech industry relies on a new shiny bubble. It is no longer crypto or NFTs. Investors are losing faith in big tech companies, interest rates are up, layoffs are happening on a large scale and people are not as locked up in their houses as they were 3 years ago. Investors are now looking at other industries like agriculture and energy. This is why the GPT hype is a move by tech companies to make the world feel like this new upcoming “sentient AGI” is here to replace jobs, cut down redundancy and maximise profit in the tech industry and beyond. It is basically a way to convince investors that their investments in tech are safe.
chatGPT is useful in some cases
No need for having tabs of documentation or solving a problem that has been solved a million times over and over. ChatGPT works very well for boilerplate code, documentation, etc. It helped me solve several basic problems in react native without me having to browse the documentation. Also, Given that C++ in Unreal engine has a bad reputation for poor documentation, GPT can help a lot of beginners get into C++ programming with Unreal engine.
It can never replace a programmer
Like I said, GPT lacks the ability to think and cannot see the big picture. Give it any application with 2 thousand lines of code and 10 different classes, then chatGPT starts to fall apart. In fact, most of the above examples I gave are solutions that could all be implemented in less than 1000 lines of code. chatGPT did not solve any of them. How long will chatGPT take to be able to write an entire software with a code base of tens or hundreds of thousand lines? Software like a payment gateway, a messaging system, etc. How long will it then take to build ridiculously complex applications like Excel, FL Studio, Cubase or Visual Studio which have millions and millions of lines of code . It will never get to this level.
So this is my take. All you junior programmers have a lot more to worry about than chatGPT. So many junior programmers I am currently mentoring have been messaging me and have been really worried. Yes, there will be more layoffs in tech. In fact, expect tens of thousand more devs to lose their jobs, but that will have nothing to do with chatGPT, it will have more to do with inefficiencies and bullshit redundant jobs in the tech industry. The tech industry needs a reset. I covered that problem in this article.