Walkthrough of using private assets

[arsian]

How would one go about using assets from their own private bucket (and not the public one in the case of generativeai-downloads/images/scones.jpg)?

Is your feature request related to a problem? Please describe.

I am facing errors (500 internal server) when attempting to access an image in my own GCS and there seems to be an access/permission issue even though I have tried to instantiate a Storage object with my application_default_credentials in order to access the files inside the bucket, and to then use the file to compose a fileUri (i.e. gs://bucket-name/filename.jpeg).

Describe the solution you'd like.

I would like to be able to provide private assets from GCS to VertexAI and get the same result as described here
https://cloud.google.com/vertex-ai/docs/generative-ai/start/quickstarts/quickstart-multimodal#send-request

1. I created a bucket
2. Then placed an image inside the bucket
3. Populated the code below with above info

const { Storage } = require('@google-cloud/storage');
const { VertexAI } = require('@google-cloud/vertexai');

const PROJECT_ID = process.env.PROJECT_ID;
const bucketName = 'arsia-toki-bucket-standard';

async function createNonStreamingMultipartContent(
    projectId = PROJECT_ID,
    location = 'us-central1',
    model = 'gemini-pro-vision',
    mimeType = 'image/jpeg'
) {
    const storage = new Storage({ projectId, keyFilename: './application_default_credentials.json' });

    // Initialize Vertex with your Cloud project and location
    const vertexAI = new VertexAI({ project: projectId, location: location });

    // Instantiate the model
    const generativeVisionModel = vertexAI.preview.getGenerativeModel({
        model: model,
    });

    const [files] = await storage.bucket(bucketName).getFiles();
    if (files.length === 0) {
        throw new Error('No files found in the bucket.');
    }

    const fileUri = `gs://${bucketName}/${files[0].name}`;

    // Create the request
    // For images, the SDK supports both Google Cloud Storage URI and base64 strings
    const filePart = {
        fileData: {
            fileUri: fileUri,
            mimeType: mimeType,
        },
    };
    const textPart = {
        text: 'what is shown in this image?',
    };

    const request = {
        contents: [{ role: 'user', parts: [filePart, textPart] }],
    };

    console.log('Prompt Text:');
    console.log(request.contents[0].parts[1].text);


    console.log('Non-Streaming Response Text:');
    // Create the response stream
    const responseStream = await generativeVisionModel.generateContentStream(request);

    // Wait for the response stream to complete
    const aggregatedResponse = await responseStream.response;

    // Select the text from the response
    const fullTextResponse = aggregatedResponse.candidates[0].content.parts[0].text;

    console.log(fullTextResponse);

    // [END aiplatform_gemini_get_started]
}

createNonStreamingMultipartContent(...process.argv.slice(2)).catch(err => {
    console.error(err.message);
    process.exitCode = 1;
});

Describe alternatives you've considered.

So one solution that I did conjure up was to add to the acl rules before sending the asset to VertexAI and once the analysis is complete, to revoke the public acl rule. But I wonder if you have a more elegant solution.

const { Storage } = require('@google-cloud/storage');
const { VertexAI } = require('@google-cloud/vertexai');

const PROJECT_ID = process.env.PROJECT_ID;
const bucketName = 'arsia-toki-bucket-standard';

async function createNonStreamingMultipartContent(
    projectId = PROJECT_ID,
    location = 'us-central1',
    model = 'gemini-pro-vision',
    mimeType = 'image/jpeg'
) {

    // Initialize Vertex with your Cloud project and location
    const vertexAI = new VertexAI({ project: projectId, location: location });

    // Instantiate the model
    const generativeVisionModel = vertexAI.preview.getGenerativeModel({
        model: model,
    });

    const storage = new Storage({ projectId: PROJECT_ID, keyFilename: './key.json' });

    // Get the file from the storage bucket
    const [files] = await storage.bucket(bucketName).getFiles();
    if (files.length === 0) {
        throw new Error('No files found in the bucket.');
    }

    // the second file is what I want the AI to analyze this time 
    const file = files[1];
    const fileUri = `gs://${bucketName}/${file.name}`;
    console.log('File URI:', fileUri);

    try {
        await file.acl.add({
            entity: 'allUsers',
            role: 'READER',
        });
        console.log('File is temporarily public.');

        // Create the request
        // For images, the SDK supports both Google Cloud Storage URI and base64 strings
        const filePart = {
            fileData: {
                fileUri: fileUri,
                mimeType: mimeType,
            },
        };
        const textPart = {
            text: 'what food items are shown in this image?',
        };
        const request = {
            contents: [{ role: 'user', parts: [filePart, textPart] }],
        };
        console.log('Prompt Text:');
        console.log(request.contents[0].parts[1].text);
        console.log('Non-Streaming Response Text:');
        
// Create the response stream
        const responseStream = await generativeVisionModel.generateContentStream(request);

        // Wait for the response stream to complete
        const aggregatedResponse = await responseStream.response;

        await file.acl.add({
            entity: 'allUsers',
            role: 'READER',
        });

        // Select the text from the response
        const fullTextResponse = aggregatedResponse.candidates[0].content.parts[0].text;

        console.log(fullTextResponse);

    } finally {
        try {
            await file.acl.remove({
                entity: 'allUsers',
                role: 'READER',
            });

            console.log('Public access revoked. File is private again.');
        } catch (error) {
            console.error('Error revoking public access:', error.message);
        }
    }
}

createNonStreamingMultipartContent(...process.argv.slice(2)).catch(err => {
    console.error(err.message);
    process.exitCode = 1;
});

Additional context.

Thanks!

[TokiUrata]

I'm encountering the same 500 internal server error accessing private GCS assets in VertexAI. Tried the code with proper credentials, but still facing permission issues. Any insights or solutions?

Appreciate any help!