My Experiments With ChatGPT For Software Development

When Open.AI launched ChatGPT in November 2022, people were hardly curious about how this new AI tool works. But then, with proven results, there curiosity rose. And at one point, they became anxious about the future and how it will affect humanity. I had a completely different take. I just wanted to know how we can benefit from it and so I tried using ChatGPT for software development. 

And the results are optimistic for the CTOs, tech leaders and product managers who want to increase the productivity of their development teams. ChatGPT becomes a very powerful tool with the right set of prompts and proper guidance of an experienced developer. It is true for the entire product development cycle.

Developing software majorly involves designing, coding, testing, and deployment. My objective was to determine whether developers can use ChatGPT to accelerate the software development. 

I kept my experiment simple. 

I wanted to develop an application with notification functionality for a social media app. But the experiment covered almost all phases of software development, from database design and feature code writing to documentation, unit testing, and deployment. 

For better understanding, I have shared all the chat links in each step. At the end of the article, I have also mentioned the pros, cons and suggestions regarding usage of ChatGPT for software development.

Note: I conducted the experiment in May 2023. As the AI model is evolving frequently, the results may vary in the future. Additionally, the results may differ for tools and technologies other than those used in this experiment. 

You may also interested In: Implementation of AI

Let’s start with the Database design.

Database Design with the Help of ChatGPT:

I started by designing a database schema for notification functionality using ChatGPT and generate entity classes from the suggested schema. As tools I used-

• Postgresql 
• Java 
• Spring Data JPA (Used for generate entities from the DB schema) 

Chat – https://sharegpt.com/c/hwCkEpM 

Once done, I realized that ChatGPT can make the difference in several ways. 

First, it is proficient in designing database schemas. Most of the schemas that ChatGPT is delivering are accurate. But it requires manual review, as some columns may not be necessary or are missing. Also, users must continuously interact with ChatGPT to resolve the issues in the responses.

ChatGPT can generate queries for inserting, updating, and other operations to simulate the app’s flow. I encountered no errors in the queries. It also generated an accurate mapping of tables to Spring JPA entities. The Spring Boot server started without any errors.

Time Estimation comparison:

1. Developer – 8 hrs
2. ChatGPT + developer – 2 hrs

Creating REST APIs Using ChatGPT:

I created REST APIs to access the data from the database schema I had built earlier using ChatGPT. As tools, I used- 

• Java 
• Spring boot 
• Postgresql 

Chat – https://sharegpt.com/c/hb18r0F

Refer to interaction after the question “Considering the above database design and the JPA entities, can you create API to get posts of a user?” 

While working with the DTO, I realized that it was missing some required fields and had few extra fields. The mapper class also revealed some errors like an unmapped field or incorrect mapping. 

The business logic was mixed with the controller layer. But once I made ChatGPT aware of the inappropriateness of mixing business logic with the controller layer, it resolved the issue. 

You may have to go for multiple interactions with ChatGPT to create a functional code. For instance, you may need to ask specifically for the generation of a pagination API or the use of a different library. 

Time Estimation Comparison: 

1. Developer – 8 hrs 
2. ChatGPT + developer – 5 hrs 

Creating Deployment Scripts with help of ChatGPT:

I used ChatGPT to create DockerFile, Docker Compose files for our application. Then, I used it to create a GitHub workflow file. 

DockerFile & Docker Compose

Chat – https://sharegpt.com/c/TPkbmV8

I quickly realized that ChatGPT cannot find the tool and technologies used in a project. It requires a detailed info about the tools and technologies used in the project to generate docker files. And the prompts have to be correct and precise. 

You may have to modify these files as per the application. For example, I had to change the path to the jar files. 

When it comes to finding errors in the generated files, developers have to interact with the ChatGPT to get issues fixed. In my project, ChatGPT used a generic name for the docker image in the docker compose file instead of the name provided in the DockerFile. I had to manually verify them for better implementation.

Time Estimation Comparison: 

1. Developer – 8 hrs 
2. ChatGPT + developer – 5 hrs

Generating Github Workflow Files Using ChatGPT: 

Chat – https://sharegpt.com/c/0iusgMN 

I feel confident about ChatGPT’s ability to generate GitHub workflow files. In my project, it did so based on our requirements. For example, it generated a workflow file with the configuration that triggers the pipeline on a pull request or when code is pushed to the main branch. It also provides configuration steps for the build and test actions. 

However, the versions of the plugins that ChatGPT shared were not the latest ones, which resulted in few errors that I had to analyze and fix. 

For code generation, you need precise steps. To deploy, you should first upload the artifact, copy the artifact to EC2, and start the service. If we do not provide a precise problem statement, ChatGPT may generate a different and potentially suboptimal way to deploy the application. For example, it may provide a way to clone and build the repository on EC2 instead of using the artifacts for deployment on EC2. 

Time Estimation Comparison: 

1. Developer – 4 hrs
2. ChatGPT + developer – 2 hrs

Unit Tests Findings 

Java: Most of the generated test cases were correct. 
Chat – https://sharegpt.com/c/1AtliX8 

Rust: ChatGPT generated the test cases but there were errors in the generated code. 
Chat – https://sharegpt.com/c/8fMkM29
https://sharegpt.com/c/UDmroBA

Pros and Cons of Using ChatGPT for Software Development:

Here’s what I have realized while using ChatGPT for software development-

Pros: 

1. ChatGPT can help developers with generating the boilerplate code quickly. 
2. It works well for generating database design and SQL queries. It can generate the mapping of tables to code precisely. 
3. Its performance is laudable when it comes to generating deployment files. It can generate most of the parts of the Docker, Docker Compose, and GitHub Workflow files. 
4. Generates good documentation of the code snippets. 
5. ChatGPT makes your chat interactive- it is a plus for any generative AI tool.

Cons: 

1. Developers must interact with it multiple times to get the expected output. 
2. Sometimes, ChatGPT loses the context and the developers need to be thorough with their prompts.
3. There can be errors, such as missing library, outdated library code, etc. which the developer has to fix manually. Or, he can take help from ChatGPT to fix those with new prompts. 
4. ChatGPT stops the answer in between if the answer is big. 
5. ChatGPT is not trained on the recent data. So, we cannot use it for the latest tool or the latest versions of tools/libraries. But this will improve with the future updates.
6. Does not work well with complex functionalities. It itself suggests referring to the official documentation. 

You may also interested In: Conversational AI: Top Challenges You Must Avoid

Suggestions:

1. Be detailed in the prompts and provide examples to set up the context.
2. Use it to generate small code snippets.
3. Verify the code manually. You have to be careful about the code snippet quality. You must also verify the library version, and check for missing fields, extra fields, etc. 
4. If you inform ChatGPT that a particular part of the code snippet is incorrect, the tool will most likely correct it if you use the right prompts.
5. The development process will consume less time if the developer has a high-level knowledge of the solutions to the problems.  So, it favours the skilled developers.

Point to note: At such a nascent stage, ChatGPT can raise some red flags regarding IP issues. If the product is security sensitive, then entrepreneurs should adopt ChatGPT judiciously and should review the code thoroughly.

Final Thoughts: 

Product teams often work in tight deadlines and ChatGPT could decrease their feature shipment timelines drastically. What I have experienced in my project, ChatGPT is good when it comes to generating database schemas, SQL queries, Docker files, GitHub workflow files, and documentation. But only if there is someone to supervise the entire process. A healthy teaming up of developers and the AI tool is the way to go.

Ready to explore personalized AI development services? Our AI solutions, making a daily impact on millions, are backed by expert AI development teams. Comprising skilled professionals in AI, data science, development, UI/UX, and DevOps, our teams have successfully executed over 30 challenging AI implementations.
Get in touch with our AI expert now!