Provision ICD Elasticsearch in the Gen AI RAG DA Stack

[ajainghub]

The RAG DA Stack should provision ICD Elasticsearch.

The Elasticsearch instance provisioning will require some input ( edition, version, sizing, encryption etc.). Upon provisioning the Elasticsearch instance information (host, port, private and public url, admin credentials, certificate etc.) must be provided as DA outputs. These output values will be used to configure integration in other services and DAs that use this Elasticsearch instance.

Here are some initial high-level requirements:
• Provision Elasticsearch of a selected edition, version, size,
• If own key encrypt is selected, integrate with Key Protect - create/identify key use for encrypt (requires service cross authorization)
• Add ELSER model support - ELSER 1 and 2
• Create ES admin user password, get certificates etc. (generate and set a new random password)
• Add credentials to Secrets Manager for use by other DAs
• Set DA and Stack level outputs
• Option to deploy Kibana (on Code Engine) and connect the Elasticsearch instance
o Add Kibana credentials to Secrets Manager

Input: (draft)
o IBM Cloud Admin API Key (and IAM Bearer Token)
o Region, ES Name, Resource Group
o Database Edition/Plan (Default – Enterprise/Platinum TBD)
o Version (Default - 8.12)
o Sizing
Default for Enterprise
• Small. 0.5 vCPUs; 4 GB RAM; 5GB Disk
• Hosting model - Shared
Default for Platinum
• XS 4x16
• Hosting model - Isolated
o If customer managed encryption Y
Key Protect Key (from Key Protect)
o Endpoints – Both Public and Private

Output: (draft)
o ICD ES Deployment ID
o ES Hostname
o ES Port
o ES Version
o ES Edition
o ES Public Endpoint and port
o ES Private Endpoint and port
o ES admin user password (in Secrets Manager)
o ES certificate (in Secrets Manager)
o Kibana URL