Skip to content

GCP Vertex AI Integration

Architecture Overview

Purpose: Enable AI-powered contact center analytics for WxCC without building on-premise ML infrastructure.

Connectivity Model: Internet-based (no dedicated circuits)

Rationale: 1. WxCC is already in Cisco Cloud (not on-premise) 2. Data transfer: WxCC Cloud → GCP Cloud (cloud-to-cloud, Cisco backbone) 3. On-premise access: Admins/data scientists access via Umbrella SASE DIA 4. Volume: ~6GB/day (CDR, recordings, transcripts) - doesn't justify $4,500/month Cloud Interconnect

┌─────────────────────────────────────────────────────────────────────────────┐
│               GCP VERTEX AI CONNECTIVITY (INTERNET-BASED)                    │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                              │
│  ┌─────────────────────────────────────────────────────────────┐           │
│  │ WEBEX CONTACT CENTER (CISCO CLOUD - US REGION)              │           │
│  │ ─────────────────────────────────────────────────────────   │           │
│  │ • 175 Agents                                                 │           │
│  │ • ~5,000 Calls/Day                                           │           │
│  │ • Call Recordings, CDR, Agent Metrics                        │           │
│  └────────────────────┬────────────────────────────────────────┘           │
│                       │                                                      │
│                       │ ① Cloud-to-Cloud Transfer                          │
│                       │    (Cisco → GCP Backbone)                          │
│                       │    No internet involved                             │
│                       │                                                      │
│                       ▼                                                      │
│  ┌─────────────────────────────────────────────────────────────┐           │
│  │ GCP VERTEX AI (us-east4 Region - Virginia)                  │           │
│  │ ─────────────────────────────────────────────────────────   │           │
│  │ ② Data Ingestion:                                           │           │
│  │    • WxCC Webhook → Pub/Sub (real-time)                     │           │
│  │    • Call Recordings → Cloud Storage                        │           │
│  │    • CDR Export → BigQuery (every 5 min)                    │           │
│  │                                                              │           │
│  │ ③ AI Processing:                                            │           │
│  │    • Vertex AI Speech-to-Text: Transcribe calls             │           │
│  │    • Vertex AI NLU: Sentiment analysis                      │           │
│  │    • CCAI Insights: Call analytics dashboard                │           │
│  │    • Custom Models: Churn prediction, routing               │           │
│  │                                                              │           │
│  │ ④ Output:                                                   │           │
│  │    • Predictions → WxCC API (route calls)                   │           │
│  │    • Dashboards → Looker Studio (agent performance)         │           │
│  │    • Alerts → Splunk (quality issues)                       │           │
│  └─────────────────────────────────────────────────────────────┘           │
│                       ▲                                                      │
│                       │                                                      │
│                       │ ⑤ Admin Access (Mumbai)                            │
│                       │    Via Umbrella SASE DIA                            │
│                       │    TLS 1.3 Encrypted                                │
│                       │                                                      │
│  ┌────────────────────┴────────────────────────────────────────┐           │
│  │ ON-PREMISE USERS (DATA SCIENTISTS, ADMINS)                  │           │
│  │ ─────────────────────────────────────────────────────────   │           │
│  │ Access: https://console.cloud.google.com                    │           │
│  │         https://wxcc-ml-dashboard.abhavtech.com             │           │
│  │                                                              │           │
│  │ Route: User → SD-WAN Edge → Umbrella SASE → Internet → GCP │           │
│  │ Security: Duo MFA + Azure AD SSO + TLS 1.3                  │           │
│  └─────────────────────────────────────────────────────────────┘           │
│                                                                              │
└─────────────────────────────────────────────────────────────────────────────┘

GCP Services Architecture

Core Services (With Critical Security Enhancements)

Service Purpose Security Enhancement
Contact Center AI (CCAI) Platform Purpose-built WxCC analytics ✅ VPC Service Controls perimeter
Agent Assist Real-time agent coaching ✅ Private Service Connect
Dialogflow CX IVR NLU optimization ✅ VPC Service Controls
Vertex AI Speech-to-Text Call transcription ✅ DLP API (PII redaction)
Vertex AI NLU Sentiment analysis ✅ DLP API (PII redaction)
BigQuery (WxCC Dataset) CDR, metrics storage ✅ Column-level encryption (CMEK)
Cloud Storage (Recordings) 90-day retention ✅ Bucket-level encryption (CMEK)
Vertex AI Custom Training Churn/CSAT models ✅ Model Monitoring enabled
Vertex AI Feature Store Centralized features ✅ Access controls (IAM)
Cloud DLP PII redaction ✅ CRITICAL for PCI-DSS

Critical Security Services (DEEP DIVE)

1. VPC Service Controls (Perimeter Security)

Purpose: Create security perimeter around WxCC-sensitive GCP resources to prevent data exfiltration.

Architecture:

┌─────────────────────────────────────────────────────────────────────────────┐
│                    VPC SERVICE CONTROLS PERIMETER                            │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                              │
│  ┌─────────────────────────────────────────────────────────────────┐       │
│  │ TRUSTED ZONE (Inside Perimeter)                                 │       │
│  │ ─────────────────────────────────────────────────────────────   │       │
│  │                                                                  │       │
│  │  ┌──────────────────┐  ┌──────────────────┐  ┌───────────────┐│       │
│  │  │ Vertex AI        │  │ BigQuery         │  │ Cloud Storage ││       │
│  │  │ (Speech, NLU,    │  │ (wxcc_analytics) │  │ (recordings)  ││       │
│  │  │  Training)       │  │                  │  │               ││       │
│  │  └──────────────────┘  └──────────────────┘  └───────────────┘│       │
│  │                                                                  │       │
│  │  INGRESS POLICY:                                                │       │
│  │  ✅ Allow from: WxCC Cloud IPs (52.x.x.x/16)                   │       │
│  │  ✅ Allow from: Abhavtech Admin IPs (10.252.0.0/16 via NAT)    │       │
│  │  ❌ Block: All other public internet sources                   │       │
│  │                                                                  │       │
│  │  EGRESS POLICY:                                                 │       │
│  │  ✅ Allow to: WxCC Webhook API (webhook.wxcc-us1.cisco.com)    │       │
│  │  ✅ Allow to: Splunk On-Prem (10.252.100.50 via Cloud VPN)     │       │
│  │  ❌ Block: All other destinations (prevents data exfiltration) │       │
│  │                                                                  │       │
│  └─────────────────────────────────────────────────────────────────┘       │
│                                                                              │
│  ❌ BLOCKED SCENARIOS:                                                      │
│  ─────────────────────────────────────────────────────────────────          │
│  • Compromised VM tries to copy recordings to attacker's S3 bucket → BLOCKED│
│  • Malicious script tries to exfiltrate BigQuery data to pastebin → BLOCKED│
│  • Admin accidentally misconfigures public bucket → BLOCKED by policy       │
│                                                                              │
└─────────────────────────────────────────────────────────────────────────────┘

Configuration (Terraform):

# VPC Service Controls Perimeter

resource "google_access_context_manager_service_perimeter" "wxcc_perimeter" {
  parent = "accessPolicies/${var.access_policy_id}"
  name   = "accessPolicies/${var.access_policy_id}/servicePerimeters/wxcc_data"
  title  = "WxCC Sensitive Data Perimeter"

  status {
    restricted_services = [
      "bigquery.googleapis.com",
      "storage.googleapis.com",
      "aiplatform.googleapis.com",
      "speech.googleapis.com",
      "language.googleapis.com"
    ]

    resources = [
      "projects/123456789"  # abhavtech-wxcc-prod project
    ]

## INGRESS: Who can access resources inside perimeter

    ingress_policies {
      ingress_from {
        sources {
          access_level = google_access_context_manager_access_level.wxcc_source.id
        }
        identity_type = "ANY_IDENTITY"
      }
      ingress_to {
        resources = ["*"]
        operations {
          service_name = "bigquery.googleapis.com"
          method_selectors {
            method = "*"
          }
        }
      }
    }

## EGRESS: Where can data go from inside perimeter

    egress_policies {
      egress_from {
        identity_type = "ANY_IDENTITY"
      }
      egress_to {
        resources = ["projects/987654321"]  # Allow to Splunk integration project only
        operations {
          service_name = "storage.googleapis.com"
          method_selectors {
            method = "google.storage.objects.get"
          }
        }
      }
    }
  }
}

## Access Level (Who can access)

resource "google_access_context_manager_access_level" "wxcc_source" {
  parent = "accessPolicies/${var.access_policy_id}"
  name   = "accessPolicies/${var.access_policy_id}/accessLevels/wxcc_sources"
  title  = "WxCC Trusted Sources"

  basic {
    conditions {
      ip_subnetworks = [
        "52.0.0.0/8",        # WxCC Cloud IP ranges
        "203.0.113.0/24"    # Abhavtech Cloud NAT (for admin access)
      ]
    }
  }
}

Compliance Impact: ✅ Satisfies PCI-DSS requirement 1.3 (network segmentation)


2. Private Service Connect (Private API Access)

Purpose: Access GCP APIs over private RFC 1918 IPs instead of public internet.

Before (Public API Access):

User/WxCC → api.google.com (Public IP: 142.250.x.x) → GCP Service
Security: TLS 1.3, but DNS/routing over public internet

After (Private Service Connect):

User/WxCC → api.p.googleapis.com (Private IP: 10.200.100.10) → GCP Service
Security: Traffic never leaves Google backbone, private routing

Implementation:

## Create Private Service Connect endpoint

resource "google_compute_global_address" "private_service_connect" {
  name         = "psc-vertex-ai"
  address_type = "INTERNAL"
  purpose      = "PRIVATE_SERVICE_CONNECT"
  network      = google_compute_network.wxcc_vpc.id
  address      = "10.200.100.10"
}

resource "google_compute_global_forwarding_rule" "psc_forwarding_rule" {
  name                  = "psc-vertex-ai-rule"
  target                = "vpc-sc"
  load_balancing_scheme = ""
  ip_address            = google_compute_global_address.private_service_connect.id

## Map all GCP AI APIs to this private IP

  service_attachment    = "projects/cloud-aiplatform/regions/us-east4/serviceAttachments/all"
}

DNS Configuration (Cloud DNS Private Zone):

## Private DNS zone for Google APIs

api.googleapis.com         → 10.200.100.10 (Private)
aiplatform.googleapis.com  → 10.200.100.10 (Private)
speech.googleapis.com      → 10.200.100.10 (Private)
language.googleapis.com    → 10.200.100.10 (Private)

## Public APIs remain unchanged

www.google.com             → 142.250.x.x (Public)

Benefit: - ✅ API traffic never traverses public internet - ✅ Reduced latency (~5ms improvement) - ✅ No exposure to DDoS attacks on public Google IPs


3. Cloud Data Loss Prevention (DLP) - PII Redaction

Purpose: Automatically detect and redact PII from call transcripts before storing in BigQuery or exporting to Splunk.

PII Types Detected:

PII Type Example Redaction Method
Credit Card "4111-1111-1111-1111" [CREDIT_CARD] or Last 4 digits only
SSN "123-45-6789" [US_SOCIAL_SECURITY_NUMBER]
Phone Number "+91-98765-43210" [PHONE_NUMBER]
Email "customer@example.com" [EMAIL_ADDRESS]
Person Name "John Smith" [PERSON_NAME] or First name only
Address "123 Main St, Mumbai" [STREET_ADDRESS]
Date of Birth "1985-03-15" [DATE_OF_BIRTH]

Implementation Flow:

┌─────────────────────────────────────────────────────────────────────────────┐
│                    CLOUD DLP REDACTION PIPELINE                              │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                              │
│  ① Call Recording → Vertex AI Speech-to-Text                                │
│     Output: Raw transcript                                                   │
│     "Hi, my name is John Smith and my credit card is 4111-1111-1111-1111"  │
│                                                                              │
│                 ▼                                                            │
│                                                                              │
│  ② Cloud DLP API Inspection                                                 │
│     ┌──────────────────────────────────────────────────────────┐           │
│     │ DLP Inspection Config:                                    │           │
│     │ • PERSON_NAME: Confidence > 80%                           │           │
│     │ • CREDIT_CARD_NUMBER: Luhn algorithm validation           │           │
│     │ • Custom regex: Abhavtech customer ID format (ABV-\d{6}) │           │
│     └──────────────────────────────────────────────────────────┘           │
│                                                                              │
│                 ▼                                                            │
│                                                                              │
│  ③ De-identification Transformation                                         │
│     Method: REPLACE_WITH_INFO_TYPE (replace PII with placeholder)           │
│     Output: "Hi, my name is [PERSON_NAME] and my credit card is             │
│             [CREDIT_CARD_NUMBER]"                                           │
│                                                                              │
│                 ▼                                                            │
│                                                                              │
│  ④ Store Redacted Transcript                                                │
│     ┌──────────────────────────────────────────────────────────┐           │
│     │ BigQuery Table: wxcc_analytics.call_transcripts          │           │
│     │ ──────────────────────────────────────────────────────   │           │
│     │ call_id     | transcript_redacted           | pii_found │           │
│     │ ─────────────────────────────────────────────────────────│           │
│     │ CALL-12345  | "Hi, my name is [PERSON_NAME]"| TRUE      │           │
│     └──────────────────────────────────────────────────────────┘           │
│                                                                              │
│  ⑤ Original Audio + Metadata Stored Separately (Secure Bucket)              │
│     ┌──────────────────────────────────────────────────────────┐           │
│     │ Cloud Storage: gs://wxcc-recordings-pii-restricted/      │           │
│     │ ──────────────────────────────────────────────────────   │           │
│     │ • Access: Only QA team + compliance officers             │           │
│     │ • Encryption: CMEK (customer-managed keys)               │           │
│     │ • Audit: Every access logged to Cloud Audit Logs         │           │
│     │ • Retention: 90 days, then auto-delete                   │           │
│     └──────────────────────────────────────────────────────────┘           │
│                                                                              │
└─────────────────────────────────────────────────────────────────────────────┘

Code Example (Python - Cloud Function):

from google.cloud import dlp_v2
from google.cloud import bigquery

def redact_and_store_transcript(call_id, raw_transcript):
    """
    Redact PII from call transcript before storing in BigQuery.
    """
    dlp = dlp_v2.DlpServiceClient()
    project_id = "abhavtech-wxcc-prod"

## Define PII types to detect

    inspect_config = {
        "info_types": [
            {"name": "PERSON_NAME"},
            {"name": "CREDIT_CARD_NUMBER"},
            {"name": "US_SOCIAL_SECURITY_NUMBER"},
            {"name": "PHONE_NUMBER"},
            {"name": "EMAIL_ADDRESS"},
            {"name": "STREET_ADDRESS"},
        ],
        "min_likelihood": "POSSIBLE",  # Detect even low-confidence PII
        "limits": {
            "max_findings_per_request": 0  # No limit
        }
    }

## Define redaction method

    deidentify_config = {
        "info_type_transformations": {
            "transformations": [
                {
                    "primitive_transformation": {
                        "replace_with_info_type_config": {}  # Replace with [INFO_TYPE]
                    }
                }
            ]
        }
    }

## Redact PII

    response = dlp.deidentify_content(
        request={
            "parent": f"projects/{project_id}",
            "deidentify_config": deidentify_config,
            "inspect_config": inspect_config,
            "item": {"value": raw_transcript}
        }
    )

    redacted_transcript = response.item.value
    pii_found = len(response.overview.transformation_summaries) > 0

## Store in BigQuery

    bq_client = bigquery.Client()
    table_id = f"{project_id}.wxcc_analytics.call_transcripts"

    rows_to_insert = [{
        "call_id": call_id,
        "transcript_redacted": redacted_transcript,
        "transcript_raw_gcs_uri": f"gs://wxcc-recordings-pii-restricted/{call_id}.txt",
        "pii_found": pii_found,
        "redacted_at": datetime.datetime.utcnow().isoformat()
    }]

    errors = bq_client.insert_rows_json(table_id, rows_to_insert)

    if errors:
        raise Exception(f"BigQuery insert failed: {errors}")

    return redacted_transcript

Compliance Impact: - ✅ PCI-DSS 3.2.1: Requirement 3.4 (render PAN unreadable) - ✅ GDPR Article 25: Data protection by design and by default - ✅ CCPA: Minimize collection of personal information


4. Vertex AI Explainable AI

Purpose: Understand WHY ML models make specific predictions (e.g., why route call to senior agent?).

Use Case Example:

┌─────────────────────────────────────────────────────────────────────────────┐
│                    EXPLAINABLE AI - CALL ROUTING MODEL                       │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                              │
│  SCENARIO: Customer calls WxCC, ML model routes to "Senior Agent Tier"      │
│                                                                              │
│  INPUT FEATURES:                                                             │
│  ┌───────────────────────────────────────────────────────────────┐         │
│  │ • customer_tenure: 8 years                                     │         │
│  │ • ltv_score: $125,000 (high-value customer)                   │         │
│  │ • issue_complexity: 8/10 (billing dispute)                    │         │
│  │ • previous_escalations: 3 (last 6 months)                     │         │
│  │ • sentiment_score: -0.6 (frustrated)                          │         │
│  │ • call_time: 10:30 AM (business hours)                        │         │
│  └───────────────────────────────────────────────────────────────┘         │
│                                                                              │
│  MODEL PREDICTION:                                                           │
│  ┌───────────────────────────────────────────────────────────────┐         │
│  │ Recommended Agent Tier: SENIOR (confidence: 92%)              │         │
│  └───────────────────────────────────────────────────────────────┘         │
│                                                                              │
│  EXPLANATION (Shapley Values):                                               │
│  ┌───────────────────────────────────────────────────────────────┐         │
│  │ Feature                  | Contribution to "Senior" Decision  │         │
│  │ ────────────────────────────────────────────────────────────  │         │
│  │ ltv_score ($125K)        | +35% 🔴🔴🔴🔴🔴                   │         │
│  │ previous_escalations (3) | +28% 🔴🔴🔴🔴                     │         │
│  │ sentiment_score (-0.6)   | +18% 🔴🔴🔴                       │         │
│  │ issue_complexity (8/10)  | +12% 🔴🔴                         │         │
│  │ customer_tenure (8 yrs)  | +5%  🔴                           │         │
│  │ call_time (10:30 AM)     | +2%  ▪                            │         │
│  └───────────────────────────────────────────────────────────────┘         │
│                                                                              │
│  INTERPRETATION:                                                             │
│  ──────────────────────────────────────────────────────────────────         │
│  "This call should go to a senior agent because:                            │
│   1. High LTV customer ($125K) - we can't afford to lose them               │
│   2. History of escalations (3 in 6 months) - needs experienced handling    │
│   3. Negative sentiment detected - customer is already frustrated"          │
│                                                                              │
│  USE CASES:                                                                  │
│  • Agent sees explanation on screen pop: "High-value customer, be careful!" │
│  • Compliance audit: "Why did we not route to junior agent?" → Show exp.    │
│  • Model debugging: If accuracy drops, see which features changed           │
│                                                                              │
└─────────────────────────────────────────────────────────────────────────────┘

Implementation (Python):

from google.cloud import aiplatform

## Deploy model with Explainable AI enabled

endpoint = aiplatform.Endpoint.create(
    display_name="wxcc-call-routing-model",
    explanation_metadata={
        "inputs": {
            "customer_tenure": {"input_baselines": [0], "modality": "numeric"},
            "ltv_score": {"input_baselines": [0], "modality": "numeric"},
            "issue_complexity": {"input_baselines": [5], "modality": "numeric"},
            "previous_escalations": {"input_baselines": [0], "modality": "numeric"},
            "sentiment_score": {"input_baselines": [0], "modality": "numeric"}
        },
        "outputs": {
            "recommended_tier": {"modality": "categorical"}
        }
    },
    explanation_parameters={
        "sampled_shapley_attribution": {
            "path_count": 10  # Number of feature permutations to test
        }
    }
)

## Make prediction with explanation

prediction = endpoint.explain(
    instances=[{
        "customer_tenure": 8,
        "ltv_score": 125000,
        "issue_complexity": 8,
        "previous_escalations": 3,
        "sentiment_score": -0.6
    }]
)

## Extract explanation

for explanation in prediction.explanations:
    print("Feature Attributions:")
    for feature, attribution in explanation.attributions[0].feature_attributions.items():
        print(f"  {feature}: {attribution:.2f}")

Benefit: Transparency, trust, regulatory compliance, debugging


5. Vertex AI Model Monitoring (Drift Detection)

Purpose: Detect when models degrade over time due to data distribution changes.

Monitored Metrics:

Metric Description Alert Threshold Action
Training-Serving Skew Input data distribution changed >5% divergence Re-collect training data
Prediction Drift Output distribution changed >10% shift Retrain model
Feature Attribution Drift Model relies on different features >15% change Investigate root cause
Data Quality Missing values, outliers >2% anomalies Fix data pipeline

Example Alert:

┌─────────────────────────────────────────────────────────────────────────────┐
│                    MODEL DRIFT ALERT - CHURN PREDICTION                      │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                              │
│  ⚠️  ALERT: Prediction Drift Detected                                       │
│                                                                              │
│  Model: wxcc-churn-prediction-v2.1                                           │
│  Drift Type: Prediction Distribution                                         │
│  Severity: HIGH                                                              │
│  Detected: 2025-01-18 14:30:00 UTC                                           │
│                                                                              │
│  BASELINE (Training Data - Dec 2024):                                        │
│  ┌────────────────────────────────────────────────────────────┐             │
│  │ Predicted Churn Rate: 18% (mean)                           │             │
│  │ Distribution:                                               │             │
│  │   • Low Risk (0-30%): 65% of customers                     │             │
│  │   • Medium Risk (30-70%): 25% of customers                 │             │
│  │   • High Risk (70-100%): 10% of customers                  │             │
│  └────────────────────────────────────────────────────────────┘             │
│                                                                              │
│  CURRENT (Production - Jan 2025):                                            │
│  ┌────────────────────────────────────────────────────────────┐             │
│  │ Predicted Churn Rate: 42% (mean) ⚠️ +133% increase         │             │
│  │ Distribution:                                               │             │
│  │   • Low Risk (0-30%): 38% of customers ⬇️ -27%            │             │
│  │   • Medium Risk (30-70%): 32% of customers ⬆️ +7%         │             │
│  │   • High Risk (70-100%): 30% of customers ⚠️ +200%        │             │
│  └────────────────────────────────────────────────────────────┘             │
│                                                                              │
│  ROOT CAUSE ANALYSIS:                                                        │
│  ──────────────────────────────────────────────────────────────             │
│  • Hypothesis 1: Product pricing changed (Jan 1, 2025)                      │
│    → Check: CRM data shows 15% price increase for Enterprise tier           │
│  • Hypothesis 2: Customer support quality decreased                         │
│    → Check: Average CSAT dropped from 78% → 65% in Jan                      │
│  • Hypothesis 3: Model is broken                                            │
│    → Check: Feature distributions look normal, no data pipeline errors      │
│                                                                              │
│  CONCLUSION: Real customer dissatisfaction, not model error                 │
│                                                                              │
│  RECOMMENDED ACTIONS:                                                        │
│  1. DO NOT retrain model (model is working correctly)                       │
│  2. Escalate to business team (pricing, support quality issues)             │
│  3. Implement retention campaigns for high-risk customers                   │
│  4. Monitor closely for next 2 weeks                                         │
│                                                                              │
│  AUTOMATED ACTIONS TAKEN:                                                    │
│  ✅ Alert sent to: mlops@abhavtech.com, contact-center-ops@abhavtech.com   │
│  ✅ Slack notification: #wxcc-ai-monitoring                                 │
│  ✅ ServiceNow ticket: INC0098765 (P2 - High)                               │
│                                                                              │
└─────────────────────────────────────────────────────────────────────────────┘

Configuration (Terraform):

resource "google_vertex_ai_endpoint" "churn_model" {
  display_name = "wxcc-churn-prediction"
  location     = "us-east4"

## Enable Model Monitoring

  model_monitoring_config {
    alert_config {
      email_alert_config {
        user_emails = [
          "mlops@abhavtech.com",
          "contact-center-ops@abhavtech.com"
        ]
      }

## Slack webhook

      notification_channels = [
        "projects/123456/notificationChannels/789"
      ]
    }

## Drift Detection Thresholds

    monitoring_config {
      prediction_drift_detection_config {
        drift_thresholds = {
          "churn_probability" = 0.10  # Alert if >10% drift
        }
      }

      training_prediction_skew_detection_config {
        skew_thresholds = {
          "customer_tenure" = 0.05,
          "ltv_score" = 0.05,
          "support_tickets_30d" = 0.05
        }
      }
    }

## Monitoring frequency

    monitoring_interval_days = 1  # Check daily
  }
}

Benefit: Prevent silent model degradation, faster incident response


6. Vertex AI Feature Store (Centralized Features)

Purpose: Single source of truth for ML features, avoiding duplicate computation and inconsistency.

Problem Without Feature Store:

❌ INCONSISTENT FEATURES:

Churn Model (runs at 9 AM):
  customer_avg_call_duration_30d = 8.5 minutes  (computed from CDR)

Routing Model (runs at 2 PM):
  customer_avg_call_duration_30d = 7.2 minutes  (computed from different CDR export)

Result: Models give conflicting predictions!

Solution With Feature Store:

✅ CONSISTENT FEATURES:

Feature Store (computed once at 3 AM daily):
  customer_avg_call_duration_30d = 8.5 minutes

Both models fetch from Feature Store:
  Churn Model → Feature Store → 8.5 minutes
  Routing Model → Feature Store → 8.5 minutes

Result: Consistent predictions!

Architecture:

┌─────────────────────────────────────────────────────────────────────────────┐
│                    VERTEX AI FEATURE STORE ARCHITECTURE                      │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                              │
│  ① BATCH FEATURE COMPUTATION (Daily at 3 AM)                                │
│  ┌──────────────────────────────────────────────────────────────┐          │
│  │ Cloud Scheduler → Cloud Function → BigQuery                  │          │
│  │                                                               │          │
│  │ SQL Query:                                                    │          │
│  │   SELECT customer_id,                                         │          │
│  │          AVG(call_duration) as avg_call_duration_30d,        │          │
│  │          COUNT(*) as total_calls_30d,                        │          │
│  │          AVG(csat_score) as avg_csat_30d                     │          │
│  │   FROM wxcc_analytics.calls                                   │          │
│  │   WHERE call_date >= CURRENT_DATE - 30                       │          │
│  │   GROUP BY customer_id                                        │          │
│  └──────────────────────────────────────────────────────────────┘          │
│                 ▼                                                            │
│  ② STORE IN FEATURE STORE                                                   │
│  ┌──────────────────────────────────────────────────────────────┐          │
│  │ Vertex AI Feature Store                                       │          │
│  │ ──────────────────────────────────────────────────────────   │          │
│  │ Entity Type: customer                                         │          │
│  │                                                               │          │
│  │ Features:                                                     │          │
│  │ • avg_call_duration_30d      (DOUBLE)                        │          │
│  │ • total_calls_30d            (INT64)                         │          │
│  │ • avg_csat_30d               (DOUBLE)                        │          │
│  │ • last_escalation_date       (TIMESTAMP)                     │          │
│  │ • ltv_score                  (DOUBLE)                        │          │
│  └──────────────────────────────────────────────────────────────┘          │
│                 ▼                                                            │
│  ③ ONLINE SERVING (Real-Time Inference)                                     │
│  ┌──────────────────────────────────────────────────────────────┐          │
│  │ Call arrives → WxCC → Fetch features from Feature Store      │          │
│  │                                                               │          │
│  │ GET /v1/featurestores/customer/customer_id=12345             │          │
│  │                                                               │          │
│  │ Response (cached, <10ms):                                    │          │
│  │ {                                                             │          │
│  │   "avg_call_duration_30d": 8.5,                              │          │
│  │   "total_calls_30d": 12,                                     │          │
│  │   "avg_csat_30d": 4.2,                                       │          │
│  │   "last_escalation_date": "2025-01-10",                      │          │
│  │   "ltv_score": 125000                                        │          │
│  │ }                                                             │          │
│  └──────────────────────────────────────────────────────────────┘          │
│                 ▼                                                            │
│  ④ BOTH MODELS USE SAME FEATURES                                            │
│  ┌─────────────────────────────────┬────────────────────────────┐          │
│  │ Churn Model                     │ Routing Model              │          │
│  │ (Vertex AI Prediction)          │ (Vertex AI Prediction)     │          │
│  │                                 │                            │          │
│  │ Input: Features from store      │ Input: Features from store │          │
│  │ Output: Churn prob = 0.35       │ Output: Route to tier 2    │          │
│  └─────────────────────────────────┴────────────────────────────┘          │
│                                                                              │
└─────────────────────────────────────────────────────────────────────────────┘

Benefit: Consistency, performance (cached features), governance


GCP Summary Table

Component Purpose Security
CCAI Platform Purpose-built WxCC analytics VPC Service Controls
Agent Assist Real-time coaching Private Service Connect
Dialogflow CX IVR optimization VPC Service Controls
BigQuery CDR storage CMEK + column encryption
Cloud Storage Recordings CMEK + bucket policies
Custom Models Churn, CSAT, routing Model Monitoring
Feature Store Centralized features IAM + audit logs
Cloud DLP PII redaction Built-in
VPC Service Controls Perimeter security Critical
Private Service Connect Private API access Critical