Can't Get No Satisfaction 😀 - gpt-engineer has yet to actually deliver a project for me

[clickbrain]

First let me say, that I love where gpt-engineer is as of now in terms of how rapidly it is maturing and the community is growing. It is exciting to be involved in a small way.

Please excuse this overly long post.

TLDR: Technical non-devs will have trouble getting to a finished product using gpt-engineer as it currently is. Is the target market for this meant to be primarily developers or is there a goal of making it work for other technical non-devs in the long run?

I've posted some issues for gpt-engineer that I think make it difficult to actually get to completion on a project. As well, I've commented on some others.

I started using gpt-engineer, because I could tell how powerful it was at not only developing the specification, but also guiding to professional code and implementing testing.

I am beginning to think though, that I am not the target user for the tool due to my lack of comprehensive development experience. While I have managed a ton of devs on projects in my career and I have a solid understanding of how coding functions and what is possible, and have 3+ decades of tech experience and building, I have never been someone that took the time to actually learn to code in depth.

I have successfully complete 4-5 development projects using the ChatGPT interface as well as TypingMind. It is though an arduous process and I doubt seriously that anyone with no development understanding can survive the process.

So, I have diligently tried 4 times to build software with the gpt-engineer without success. I attribute this to a number of different things and primarily to how early it is in the life of the project, but I want to express these things, in case I will never be the target market and I shouldn't waste my time engaging with the project. If the project is primarily meant to help experienced devs more rapidly get to market with products, then I completely understand & I'll go back to my "manual" ChatGPT process.

Things that have prevented success:

1. Errors in testing process. It seems to want to start to do testing when it finishes coding, but sometimes I can't tell what it does and in the case of a web app, I had no idea how to fix the numerous problems found with its own code, so I chock this up to not being an experienced dev in order to complete the process. That said, maybe an app like this needs to focus on say just a couple of foundational technologies like Python, Node, etc. and then build in processes for testing into the app, so that the user can interact and request fixes or gpt-engineer can process everything that is found and output the final code after more rounds of testing.

2. Errors in the code - In the case of two apps using Python - one a Python app and one a Telegram bot, there were a number of errors in the code in terms of syntax, libraries, etc. I could manually go through those to try to fix the errors, but that process is so much easier with ChatGPT, so that is why I requested what I mentioned above to have an interactive process where the user can feedback errors encountered running the code, changes they want to make to features or UI, etc. and have gpt-engineer fix or change the code to meet the user's needs.

3. What's next? The app doesn't really communicate to the user what to do at the end of the process. Whether that is for testing, installing, running, etc. I had another Issue about this that I posted as well.

So, this long long post is to really ask a simple question. Am I has a highly technical non-developer the target market for this app or would I be better off investing my time in what I am already doing or another platform. Unfortunately I see two targets for most apps like this - experienced developers or dumbed down apps for complete newbs.

I actually think there is a huge market in between. If you think about all the product managers and others in the world that have tech skills, want to create applications, because they understand what is possible in a given arena, then I think a lot of folks could make great use of the product to create powerful things and help in the community with refining what it does.

Apologies for the overly long post, but hopefully it communicates the experience. Thank you @AntonOsika and everyone contributing for such an amazing start to this cool product. I am very hopeful for the future of it, but want to be sure I am the right type of user you want to have.

[patillacode]

TLDR: Technical non-devs will have trouble getting to a finished product using gpt-engineer as it currently is. Is the target market for this meant to be primarily developers or is there a goal of making it work for other technical non-devs in the long run?

I don't want to speak for the original creators but I'd say this is for developer/engineers at least for now, rather than a mainstream product. (Please read the whole thing @clickbrain there is context!)

As you know ChatGPT (the real motor of all projects similar to this one) is not perfect (yet) and if this was a product for anyone you would get an avalanche (we kind of already are under it a bit) of people saying "it doesn't work"
And surely, it might not work like they expect.

My initial impression, on people that don't really understand what AI is or specifically LLMs are, is that they believe this is a "click a button and everything works"
While that might be the end goal (shoot for the stars!) it is obvious we are not there yet and anything you create with GPT Engineer (or any similar project) needs to be a bit curated, reviewed and probably tweaked, fixed... And to do that you need a bit of a technical background.

The classic "Trust, but test" approach really applies here IMHO.

I have successfully complete 4-5 development projects using the ChatGPT interface as well as TypingMind. It is though an arduous process and I doubt seriously that anyone with no development understanding can survive the process.

I agree, I have seen people in the same situation. Hence my previous comment, you still need a bit of dev skills if you want to work with this AFAIU, although I also have to say that for small things GPT Engineer is working pretty much perfectly (have a look at the benchmarks and the results we are getting)

So, I have diligently tried 4 times to build software with the gpt-engineer without success.

Would you mind sharing the prompts you used? I mean, I have tried small things and they are pretty much working (besides small issues, file names or small hallucinations by GPT).

I'll go back to my "manual" ChatGPT process.

I highly doubt you can get better and/or faster results by hand.

I'd say GPT Engineer takes a bit to understand and work with, judging by your comments around the repo I'd say you absolutely have the skills to work with it, the learning curve might be a bit steep, no doubt, and you might need to "get your hands dirty" but why not?

Errors in the code - In the case of two apps using Python - one a Python app and one a Telegram bot, there were a number of errors in the code in terms of syntax, libraries, etc

That is not really an issue with GPT Engineer, bare in mind those issues are more ChatGPT related (especially if you don't have access to the GPT-4 model)

want to be sure I am the right type of user you want to have.

I think we want to have any user that wants to use GPT Engineer, but specially at the beginning developers will have a bit of an advantage.

I understand your frustration, really, I would suggest you push a bit, maybe learn a bit of coding while you try things out and don't give up 😄

Even if you are just "not gonna learn" for whatever reason (this is a great opportunity for it but you might not have the time) I believe your view could be helpful.

I am sure @AntonOsika (and all collaborators) could use people like you, not as technical but with ideas and vision over other things we devs cannot grasp.

You can still be a great addition to the community even if you don't code.

Thanks @clickbrain

[clickbrain]

I highly doubt you can get better and/or faster results by hand.

Admittedly a couple of the other projects I tried in gpt-engineer were more complex in nature and may just be too difficult for where gpt-engineer is today.

Example(1) of more complex: Telegram Bot to function as Task Manager, Journal, Notes

This one failed, because of several errors in the generated code as well as some dependency issues. Not having a way to send back the errors to gpt-engineer and to ChatGPT, makes it difficult to troubleshoot.

While admittedly, I get a much more organized process for this in gpt-engineer, in ChatGPT itself, I can just copy and paste the errors and have it provide fixes to the code.

A Telegram bot that manages my tasks and notes using hashtags.

Needs a UI in Telegram with buttons, etc. for interacting with the bot,but also needs to operate with slash commands.

I am using Python3 on my Mac. Be sure that I will be able to run it and setup using Python3. 

Needs to be able to search for specific types of notes, hashtags, or dates within the bot group, because it keeps track of the hashtags and 
makes them available as buttons.

Tasks must have Due Dates as well as Start and End dates and they must have a classification for Company_Name/Project_Name.

There needs to be an easy way to add tasks and notes.

All notes must be stored by date and time and can have tags associated with them. 

The bot must recognize links and store them.

It should recognize markdown links and formatting where possible in Telegram. 

I would like pre-built reports with buttons that show tasks that start or are due that day. 

Users will need an easy way to mark tasks complete with checkboxes and/or slash commands.

It would be great to have a way to access multiple google calendars in the interface and pull up my schedule and then add new appointments as well as get appointment alerts. 

Needs be a command for accessing ChatGPT in the bot and way to assign tasks to ChatGPT and have it work on them as an autonomous agent.

I must have full step by step instructions for setting up every aspect of this bot easily and cheaply on a server to make it accessible and how to setup the bot in Telegram.

I did not test this one in manual ChatGPT, but may do that a bit later, but I know that in ChatGPT, I can go back in and have it fix problems, make changes, etc.

Example (2) less complex. A short sale trading signal in Pinescript for TradingView.

I first ran this in gpt-engineer. Much easier with a single file language like Pinescript to test the process.

With gpt-engineer, I got a bunch of different errors when saving the generated code in TradingView and even though I added the errors to Main_Prompt and repeatedly asked it to fix when I reran the job multiple times, it could not get around it.

When I entered the exact same prompt in CHatGPT, I got a different set of errors, but it was able to fix it immediately to create a working version and then I could go back to ChatGPT to ask it to add functions, features, remove things, etc.

So, you can see how in a simple version, I still have more ability to fix things in the manual process, but I can't seem to get gpt-engineer to get ChatGPT to fix the errors when running there.

It may be that what I am expecting to be produced differs greatly in gpt-engineer because of the testing routines. It would be great to have a process of revisiting these and then updating with errors, changes, etc. like an ongoing project management process, so that the user doesn't have to repeat everything over and over again. I am afraid to look at my ChatGPT usage levels.

That is not really an issue with GPT Engineer, bare in mind those issues are more ChatGPT related (especially if you don't have access to the GPT-4 model)

I do have GPT-4 and I understand the errors in the code are not gpt-engineer's issue, but being unable through the interface to address the issues means that I either give up or go back to the chatgpt interface.

I understand your frustration, really, I would suggest you push a bit, maybe learn a bit of coding while you try things out and don't give up 😄

Even if you are just "not gonna learn" for whatever reason (this is a great opportunity for it but you might not have the time) I believe your view could be helpful.

Unfortunately, that just isn't in the cards right now. I don't really even have the time to be writing these big posts, but because the quality of gpt-engineer is so high, I want to try to help make it better if there is a chance it will be able to address some of the issues I am having difficulty with.

One of the web apps I tried deployed a server, but the debugging process was complex in the web interface and beyond my knowledge. I could see eventually a way for gpt-engineer to address what is found there, but that would require a web interface to it to capture everything generated in test runs.

I've actually started a prompt to build out a web interface to gpt-engineer (similar to stable-diffusion-webui) to make it easier for average folks to get up and running, but to also be a place for pushing gpt-engineer to see the outputs/errors of the code and then work on the errors, but I suspect that will be very difficult app to create and my time is tight.

I'll hang around and continue to provide feedback and use as much as I can, but will be a bit before I can actually use it effectively I think.

One more thing - documentation on how the process works and what to expect with the testing strategy would be helpful. Things like after the process runs, you can do X to fix the issues or Y to deploy.

Anyway, thanks again for the thoughtful response.

[arksenu]

I don't know how helpful this will be, but you gotta setup a virtual environment. Otherwise, you're gonna have a lot of conflicting versions of shit.

[pedro-santos21]

What types of syntax errors are you getting? Care to give us an example?