Part 3: Automated Testing for Prompts #5

imaustink · 2024-04-30T23:53:24Z

Link blog article once published.

imaustink · 2024-04-30T23:53:55Z

docker-compose.yml

+      test: ["CMD-SHELL", "curl --fail http://localhost:8233/ || exit 1"]
+      interval: 10s
+      timeout: 30s
+      retries: 3


Add health check to ensure Temporal is running.

imaustink · 2024-04-30T23:54:27Z

docker-compose.yml

+      localstack:
+        condition: service_healthy
+      postgres:
+        condition: service_started


Wait for Temporal to be healthy.

imaustink · 2024-04-30T23:54:35Z

docker-compose.yml

+      localstack:
+        condition: service_healthy
+      postgres:
+        condition: service_started


Wait for Temporal to be healthy.

imaustink · 2024-04-30T23:55:20Z

docker-compose.yml

+      sh -c '\
+        sleep 10 \
+        && npx nodemon --watch src --ext ts src/test-prompts-worker.ts \
+      '


Added worker for testPromptsWorkflow.

imaustink · 2024-04-30T23:55:35Z

package.json

@@ -6,7 +6,8 @@
    "build": "tsc --build",
    "start": "ts-node src/process-documents-worker.ts && ts-node src/invoke-prompt-worker.ts",
    "process-documents": "ts-node src/process-documents-client.ts",
-    "invoke-prompt": "ts-node src/invoke-prompt-client.ts"
+    "invoke-prompt": "ts-node src/invoke-prompt-client.ts",
+    "test-prompts": "ts-node src/test-prompts-client.ts"


Run testPromptsWorkflow.

imaustink · 2024-05-01T00:02:25Z

src/activities/invoke-prompt-activities.ts

-  }
-  return _gptModel
-}
+const getGPTModel = createMemoizedOpenAI();


Moved to central location for sharing.

imaustink · 2024-05-01T00:02:54Z

src/activities/invoke-prompt-activities.ts

+    [ 'system', 'Whenever possible, use code examples derived from the documentation provided.' ],
+    [ 'system', 'Import references must be included where relevant so that the reader can easily figure out how to import the necessary dependencies.' ],
+    [ 'system', 'Do not use your existing knowledge to determine import references, only use import references as they appear in the relevant documentation for Hatchify' ],
+    [ 'system', `Here is the Hatchify documentation that is relevant to the user's query:` + relevantDocumentation.join('\n\n') ],


Updated prompt to improve quality of results.

imaustink · 2024-05-01T00:04:05Z

src/activities/process-documents-activities.ts

-        contentColumnName: 'content',
-        metadataColumnName: 'metadata',
-      }
-    }


We close the connection after each execution so memoization leads to an error on the second execution.

imaustink · 2024-05-01T00:05:56Z

src/workflows.ts


-const { createS3Bucket, deleteS3Object, deleteS3Bucket } = proxyActivities<typeof activities>({
+const { createS3Bucket, deleteS3Object, deleteS3Bucket, generatePrompt, invokePrompt, loadTestCases } = proxyActivities<typeof activities>({


Consolidated "1 minute" activities.

imaustink · 2024-05-01T00:09:20Z

README.md

+
+- [Your Next AI Startup Should Be Built on Temporal - Part 1](https://www.bitovi.com/blog/your-next-ai-startup-should-be-built-on-temporal-part-1-document-processing)
+- [Your Next AI Startup Should Be Built on Temporal - Part 2](https://www.bitovi.com/blog/your-next-ai-startup-should-be-built-on-temporal-part-2-prompt-engineering)
+- [Your Next AI Startup Should Be Built on Temporal - Part 3]()


TODO: Update the href to point to the published article

phillipskevin · 2024-05-01T15:29:09Z

src/activities/invoke-prompt-activities.ts

+    [ 'system', 'You should respond in short paragraphs, using Markdown formatting, separated with two newlines to keep your responses easily readable.' ],
+    [ 'system', 'Whenever possible, use code examples derived from the documentation provided.' ],
+    [ 'system', 'Import references must be included where relevant so that the reader can easily figure out how to import the necessary dependencies.' ],
+    [ 'system', 'Do not use your existing knowledge to determine import references, only use import references as they appear in the relevant documentation for Hatchify' ],


phillipskevin · 2024-05-01T15:36:59Z

src/activities/test-prompts-activities.ts

+  const gptModel = getGPTModel()
+
+  const response = await gptModel.invoke([
+    [ 'system', 'You are responsible for verifying that prompt provided invoked the correct response from an LLM. Consider the following question and expected answer as well as the prompt\'s answered.' ],


should prompt's answered be prompt's answer? It likely doesn't make a difference, just curious.

phillipskevin

This looks good! Thank you for the detailed comments on the PR. I just left one comment on a possible typo. Good to merge either way.

imaustink commented Apr 30, 2024

View reviewed changes

docker-compose.yml

localstack:

condition: service_healthy

postgres:

condition: service_started

Copy link

Author

imaustink Apr 30, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wait for Temporal to be healthy.

imaustink commented Apr 30, 2024

View reviewed changes

docker-compose.yml

localstack:

condition: service_healthy

postgres:

condition: service_started

Copy link

Author

imaustink Apr 30, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wait for Temporal to be healthy.

imaustink commented Apr 30, 2024

View reviewed changes

imaustink force-pushed the part-3 branch 2 times, most recently from 4ac86c2 to b469e4b Compare May 1, 2024 00:02

imaustink commented May 1, 2024

View reviewed changes

phillipskevin reviewed May 1, 2024

View reviewed changes

phillipskevin approved these changes May 1, 2024

View reviewed changes

imaustink force-pushed the part-3 branch from b469e4b to ad96d3e Compare May 1, 2024 16:22

Part 3: Automated Testing for Prompts

c79cd42

imaustink force-pushed the part-3 branch from ad96d3e to c79cd42 Compare May 1, 2024 16:22

update README.md

a3da3ba

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Part 3: Automated Testing for Prompts #5

Part 3: Automated Testing for Prompts #5

imaustink commented Apr 30, 2024 •

edited by marshallswain

imaustink Apr 30, 2024

imaustink Apr 30, 2024

imaustink Apr 30, 2024

imaustink Apr 30, 2024

imaustink Apr 30, 2024

imaustink May 1, 2024

imaustink May 1, 2024

imaustink May 1, 2024

imaustink May 1, 2024

imaustink May 1, 2024

phillipskevin May 1, 2024

phillipskevin May 1, 2024

phillipskevin left a comment


		const { createS3Bucket, deleteS3Object, deleteS3Bucket } = proxyActivities<typeof activities>({
		const { createS3Bucket, deleteS3Object, deleteS3Bucket, generatePrompt, invokePrompt, loadTestCases } = proxyActivities<typeof activities>({

Part 3: Automated Testing for Prompts #5

Are you sure you want to change the base?

Part 3: Automated Testing for Prompts #5

Conversation

imaustink commented Apr 30, 2024 • edited by marshallswain

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

phillipskevin left a comment

Choose a reason for hiding this comment

imaustink commented Apr 30, 2024 •

edited by marshallswain