Securely Access OCI Generative AI service From application running on Oracle Cloud VMware Solution #OCVS #OCI #OCI GENAI

Generative AI is like super smart tech that can make text, images, and all sorts of stuff based on what it's learned. It helps with writing, summarizing, and suggesting things. For businesses, Generative AI can be a game-changer by making first drafts of job descriptions and product info, and giving quick summaries of articles to help customers.

Within this blog post, I am utilising secure access to OCI Generative AI for the purpose of developing an LLM-based application on the Oracle Cloud VMware solution.

Architecture

Here i have tested OCI Generative AI Playgroud Generation model which will generate Job description by providing dynamic inputs such as skills, roles, years of experience and job responsibilities. Here is reference architecture :

In this architecture i am utilising multiple OCI components for secure Access to OCI Generative AI service.

OCI Generative AI is a comprehensive managed service offering customizable large language models (LLMs). In this instance, the playground feature was utilized to experiment with the readily available pre-trained models for generating job descriptions.

OCI Functions : Here OCI Functions which have Resource principal access to OCI Generative AI service. OCI function query Generative AI service on User behalf , pass User prompt and return output.

OCI API Gateway empowers users by enabling seamless integration and communication between different services and systems. This functionality streamlines processes and enhances overall efficiency within the network.

OCI Service Gateway: OCI Service Gateway provides private access from your virtual cloud network (VCN) to other Oracle services like Object Storage and Autonomous Database without exposing them to the public internet. It helps in securing your network traffic and simplifies the connectivity setup between your VCN and Oracle services.

Steps:

First we need to set IAM policies for various OCI services work together :

For Functions to access any OCI service we should create Dynamic group :

Now we need to create policy where we will allow this Dynamic group have full access to Gen AI service :

Here i have given access to OCI function dynamic group to Generative-ai-family

Also to invoke functions using API gateway i have set policy where API gateway invoke functions.

Hey there! With OCI Generative AI, you get a bunch of ready-to-use basic models. You can pick these models in the playground to play around with different prompts and settings. Once you're happy with the results, you can simply copy and paste the code generated in the playground into your app. You can start by trying out fixed prompts in the playground and then switch them up to dynamic prompts in your code later on.

When you click on view code you will collect Endpoint and model id info. This info is required in function.

Alright, so now we gotta make an OCI Function that uses the auto-generated code from the playground.

But first, let's set up the configuration parameters:

Alright, now let's whip up a Python OCI Function named fn_genai_jobdescription.

Don't forget about the func.yaml file!

let's create func.py. In this functions i am passing prompt parameters and eventually functions will create sentence from those prompt and query GEN AI model.

import io
import json
import logging
import os

import oci.auth.signers
import oci.generative_ai
from fdk import response

try:
   endpoint = os.getenv("SERVICE_ENDPOINT")
   compartment_ocid = os.getenv("COMPARTMENT_OCID")
   model_ocid = os.getenv("MODEL_OCID")

   if not endpoint:
      raise ValueError("ERROR: Missing configuration key SERVICE_ENDPOINT")
   if not compartment_ocid:
      raise ValueError("ERROR: Missing configuration key COMPARTMENT_OCID")

   signer = oci.auth.signers.get_resource_principals_signer()
   generative_ai_client = oci.generative_ai_inference.GenerativeAiInferenceClient(config={}, service_endpoint=endpoint, signer=signer,
                                                retry_strategy=oci.retry.NoneRetryStrategy())
except Exception as e:
   logging.getLogger().error(e)
   raise
def generate_job_description(skills, role, experience, qualifications,generative_ai_client):

   generate_text_detail = oci.generative_ai_inference.models.GenerateTextDetails()
   prompt = "Generate a job description with skills {} for a role {} having experience of {}. The qualifications expected from the candidate are {}" .format(skills, role, experience, qualifications)
   prompts = prompt
   llm_inference_request = oci.generative_ai_inference.models.CohereLlmInferenceRequest()
   llm_inference_request.prompt = prompts
   llm_inference_request.max_tokens = 1000
   llm_inference_request.temperature = 1
   llm_inference_request.frequency_penalty = 0
   llm_inference_request.top_p = 0.75
   generate_text_detail.serving_mode = oci.generative_ai_inference.models.OnDemandServingMode(model_id=model_ocid)
   generate_text_detail.inference_request = llm_inference_request
   generate_text_detail.compartment_id = compartment_ocid

   try:
      generate_text_response = generative_ai_client.generate_text(generate_text_detail)
      text_response = generate_text_response.data
      return text_response.inference_response.generated_texts[0].text
   except Exception as e:
      logging.getLogger().error(e)
      raise

def handler(ctx, data: io.BytesIO = None):
   try:

      body = json.loads(data.getvalue())
      skills = body["skills"]
      role = body["role"]
      experience = body["experience"]
      qualifications = body["responsibilities"]
      job_description = generate_job_description(skills, role, experience, qualifications,  generative_ai_client)

      return response.Response(ctx, response_data=json.dumps(
                                    {"message": "please find JD {}".format(job_description)}),
                               headers={"Content-Type": "application/json"}
      )

   except Exception as handler_error:
      logging.getLogger().error(handler_error)
      return response.Response(
         ctx,
         status_code=500,
         response_data="Processing failed due to " + str(handler_error)
      )

Deploy the function, using it!

fn -v deploy --app genai

Now, let's expose the function using API Gateway. Just go ahead and create a deployment in API Gateway.

Hey, it's testing time! I've got a Linux VM up and running in the Oracle Cloud VMware Solution environment.

Here's just one example, but feel free to come up with lots of different ways to use it. It's super helpful for those old Legacy apps that can only work on VMware.