Skip to content

Week 1: Component-Level Testing

Test Matrix Week 1

┌────┬───────────────────────────────────────┬──────────────┬──────────────┐
│ ID │ Test                                  │ Platform     │ Pass Criteria│
├────┼───────────────────────────────────────┼──────────────┼──────────────┤
│1.1 │ GCP Cloud Interconnect BGP             │ GCP + C8500  │ BGP ESTAB    │
│1.2 │ GCP Vertex AI API reachability         │ GCP          │ HTTP 200     │
│1.3 │ Azure ExpressRoute BGP (3 circuits)    │ Azure + C8500│ 6/6 sessions │
│1.4 │ Azure Virtual WAN IPsec (19 sites)     │ Azure + vMgmt│ 19/19 UP     │
│1.5 │ Azure Private Endpoints DNS            │ Azure + DNS  │ 10.100.10.x  │
│1.6 │ SD-WAN fabric health (all 19 sites)    │ vManage      │ 19/19 reach  │
│1.7 │ SD-WAN BFD sessions                    │ vManage      │ >150 sessions│
│1.8 │ OMP route distribution                 │ vManage      │ All VPNs OK  │
│1.9 │ Umbrella DNS registration              │ Umbrella     │ 19/19 reg    │
│1.10│ ThousandEyes agents                    │ TE           │ 6/6 online   │
└────┴───────────────────────────────────────┴──────────────┴──────────────┘

TEST 1.1: GCP Cloud Interconnect BGP

Objective: Verify BGP sessions between Mumbai C8500-12X (MUM-HUB-01) and GCP via Dedicated Interconnect

Procedure:

! On MUM-HUB-01
show bgp ipv4 unicast summary | include 169.254.0
! Expected output:
! 169.254.0.1     4     16550    1000    900      0    0    0  1d12h     45
! 169.254.0.5     4     16550    1000    900      0    0    0  1d12h     45
! (Google BGP ASN: 16550)
! Uptime column: Should show days, not recent restart

show bgp ipv4 unicast 34.0.0.0
! Expected: Route to GCP API ranges via 169.254.0.1 (Cloud Interconnect)

! Verify prefix count from GCP
show bgp ipv4 unicast neighbors 169.254.0.1 received-routes | count
! Expected: 15-30 routes (GCP public API ranges)

Pass Criteria: - Both BGP sessions (primary 169.254.0.1 and secondary 169.254.0.5) in ESTABLISHED state - Uptime > 1 hour (stable, not flapping) - 15+ prefixes received

Failure Action: Check VLAN config on Dedicated Interconnect attachment, verify MTU (1500 minimum, 1440 for GRE overhead), check Google-assigned BGP peer IPs in Cloud Console

TEST 1.2: GCP Vertex AI API Reachability

Objective: Confirm Vertex AI API (us-central1 / asia-south1) reachable from Mumbai hub via Cloud Interconnect (not public internet)

# Run from MUM-TEST-01 (10.100.1.50)


## Step 1: Check routing — Vertex AI traffic should use Cloud Interconnect

traceroute -n aiplatform.googleapis.com
## Expected: First hops via 169.254.0.1 (Cloud Interconnect peer), NOT via public internet gateways


## Step 2: Test Vertex AI API endpoint

curl -s -o /dev/null -w "%{http_code} %{time_total}s" \
  -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  https://asia-south1-aiplatform.googleapis.com/v1/projects/abhavtech-ai/locations/asia-south1/endpoints
## Expected: 200 (or 403/404 if test project has no endpoints — that's OK, just need HTTP response)

## Latency: <10ms (Cloud Interconnect, not internet which would be 30-60ms)


## Step 3: Compare path (Cloud Interconnect vs internet)

## Via Cloud Interconnect (expected):

mtr --report -n -c 5 aiplatform.googleapis.com 2>/dev/null | tail -15
## Expected: Private hops (10.x.x.x, 169.254.x.x), final hop in Google space


## Step 4: Verify Private Service Access DNS

nslookup aiplatform.googleapis.com
## If PSA configured: Should return 10.x.x.x (private)

## If not PSA: Will return public Google IP — still OK if routing via interconnect

Pass Criteria: - Vertex AI API returns HTTP response (any 2xx/4xx — not timeout) - Traceroute shows Cloud Interconnect path (169.254.0.x hops) - Latency < 15ms from Mumbai hub

TEST 1.3: Azure ExpressRoute BGP Verification

Objective: Confirm all 6 BGP sessions (3 circuits × 2 paths) established and O365 prefixes received

#!/bin/bash

## expressroute_bgp_test.sh


echo "=== TEST 1.3: AZURE EXPRESSROUTE BGP ==="

## Expected: 6 sessions total (Mumbai, London, NJ × 2 peers each)

PASS=0; FAIL=0

check_bgp() {
  local SITE=$1
  local PRIMARY_IP=$2
  local SECONDARY_IP=$3

  echo ""
  echo "--- $SITE ---"

## Check primary BGP

  P_STATE=$(ssh admin@$SITE "show bgp ipv4 unicast summary | grep ${PRIMARY_IP}" 2>/dev/null | awk '{print $10}')
  if echo "$P_STATE" | grep -qE "^[0-9]+$"; then
    echo "  ✅ Primary (${PRIMARY_IP}): ESTABLISHED (${P_STATE} prefixes)"
    ((PASS++))
  else
    echo "  ❌ Primary (${PRIMARY_IP}): NOT established (state: $P_STATE)"
    ((FAIL++))
  fi

## Check secondary BGP

  S_STATE=$(ssh admin@$SITE "show bgp ipv4 unicast summary | grep ${SECONDARY_IP}" 2>/dev/null | awk '{print $10}')
  if echo "$S_STATE" | grep -qE "^[0-9]+$"; then
    echo "  ✅ Secondary (${SECONDARY_IP}): ESTABLISHED (${S_STATE} prefixes)"
    ((PASS++))
  else
    echo "  ❌ Secondary (${SECONDARY_IP}): NOT established (state: $S_STATE)"
    ((FAIL++))
  fi

## Check O365 prefix count (should be 500-800 per circuit)

  PREFIX_COUNT=$(ssh admin@$SITE "show bgp ipv4 unicast neighbors ${PRIMARY_IP} received-routes | count" 2>/dev/null | tail -1)
  echo "  O365 prefixes from primary: $PREFIX_COUNT"
  [[ "$PREFIX_COUNT" -ge 500 ]] && echo "  ✅ Prefix count OK (${PREFIX_COUNT} ≥ 500)" && ((PASS++)) \
    || { echo "  ❌ Low prefix count (${PREFIX_COUNT} < 500)"; ((FAIL++)); }
}

check_bgp "mum-hub-01" "169.254.200.1" "169.254.200.5"   # Mumbai (VLAN 4001)
check_bgp "lon-hub-01" "169.254.201.1" "169.254.201.5"   # London (VLAN 4002)
check_bgp "nj-hub-01"  "169.254.202.1" "169.254.202.5"   # New Jersey (VLAN 4003)

echo ""
echo "=== TEST 1.3 RESULT: $PASS PASSED, $FAIL FAILED ==="
[[ $FAIL -eq 0 ]] && echo "✅ PASS" || echo "❌ FAIL"

Pass Criteria: - All 6 BGP sessions ESTABLISHED (Microsoft ASN 12076) - 500–800 prefixes per circuit (O365 + Azure AD + Common) - LOCAL_PREF: Primary circuits = 200, secondary = 100

TEST 1.4: Azure Virtual WAN IPsec Tunnels (All 19 Sites)

#!/bin/bash

## vwan_ipsec_test.sh


echo "=== TEST 1.4: AZURE VIRTUAL WAN IPsec TUNNELS ==="

## All 19 WAN edge system IPs

SITES=(
  "mum-hub-01:10.252.1.1"   "lon-hub-01:10.252.16.1"  "nj-hub-01:10.252.32.1"
  "chn-hub-01:10.252.2.1"   "dal-hub-01:10.252.33.1"
  "blr-isr-01:10.252.50.1"  "del-isr-01:10.252.51.1"
  "hyd-br-01:10.252.52.1"   "pun-br-01:10.252.53.1"   "ahm-br-01:10.252.54.1"
  "noi-br-01:10.252.55.1"   "kol-br-01:10.252.56.1"   "jai-br-01:10.252.57.1"
  "sur-br-01:10.252.58.1"   "nag-br-01:10.252.59.1"   "chd-br-01:10.252.60.1"
  "coi-br-01:10.252.61.1"   "luc-br-01:10.252.62.1"   "fra-isr-01:10.252.17.1"
)

PASS=0; FAIL=0

for ENTRY in "${SITES[@]}"; do
  HOSTNAME=$(echo $ENTRY | cut -d: -f1)
  SYSTEM_IP=$(echo $ENTRY | cut -d: -f2)

## Check IPsec SA (IKEv2 READY state)

  SA_COUNT=$(ssh admin@$HOSTNAME "show crypto ikev2 sa | grep -c READY" 2>/dev/null)

  if [[ "$SA_COUNT" -ge 1 ]]; then
## Also check tunnel interface is up

    T_UP=$(ssh admin@$HOSTNAME "show interfaces Tunnel200 | grep -c 'line protocol is up'" 2>/dev/null)
    if [[ "$T_UP" -ge 1 ]]; then
      echo "  ✅ [$HOSTNAME]: IKEv2 READY, Tunnel200 UP"
      ((PASS++))
    else
      echo "  ⚠️  [$HOSTNAME]: IKEv2 READY but Tunnel200 DOWN"
      ((FAIL++))
    fi
  else
    echo "  ❌ [$HOSTNAME]: No IKEv2 SA (not READY)"
    ((FAIL++))
  fi
done

echo ""
echo "=== TEST 1.4: $PASS/19 sites PASS, $FAIL FAIL ==="
[[ $FAIL -eq 0 ]] && echo "✅ PASS: All 19 tunnels UP" \
  || echo "❌ FAIL: $FAIL sites need investigation"

TEST 1.5: Azure Private Endpoint DNS Resolution

#!/bin/bash

## private_endpoint_dns_test.sh


echo "=== TEST 1.5: AZURE PRIVATE ENDPOINT DNS ==="

## Internal DNS servers (on-premise AD, forward to 168.63.129.16 via conditional forwarder)

DNS_SERVER="10.252.1.100"

declare -A EXPECTED_IPS=(
  ["abhavtech-sql.database.windows.net"]="10.100.10.10"
  ["abhavtechstorage.blob.core.windows.net"]="10.100.10.20"
  ["abhavtechfiles.file.core.windows.net"]="10.100.10.21"
  ["abhavtech-kv.vault.azure.net"]="10.100.10.30"
)

PASS=0; FAIL=0

for FQDN in "${!EXPECTED_IPS[@]}"; do
  EXPECTED="${EXPECTED_IPS[$FQDN]}"

## Test from Mumbai hub (should resolve to private IP)

  RESOLVED=$(nslookup $FQDN $DNS_SERVER 2>/dev/null | grep "Address" | tail -1 | awk '{print $2}')

  if [[ "$RESOLVED" == "$EXPECTED" ]]; then
    echo "  ✅ $FQDN → $RESOLVED (private ✓)"
    ((PASS++))
  elif [[ "$RESOLVED" == 10.100.10.* ]]; then
    echo "  ⚠️  $FQDN → $RESOLVED (private, expected $EXPECTED)"
    ((PASS++))
  else
    echo "  ❌ $FQDN → $RESOLVED (expected private $EXPECTED — got public or no response)"
    ((FAIL++))
  fi
done

## Also test from branch site (Bangalore)

echo ""
echo "--- Testing from Bangalore branch (192.168.50.100) ---"
for FQDN in "abhavtech-sql.database.windows.net" "abhavtechstorage.blob.core.windows.net"; do
  RESULT=$(ssh admin@blr-user-01 "nslookup $FQDN 10.252.1.100 2>/dev/null | grep Address | tail -1" 2>/dev/null | awk '{print $2}')
  [[ "$RESULT" == "10.100.10."* ]] && { echo "  ✅ [BLR] $FQDN → $RESULT (private)"; ((PASS++)); } \
    || { echo "  ❌ [BLR] $FQDN → $RESULT (expected 10.100.10.x)"; ((FAIL++)); }
done

echo ""
echo "=== TEST 1.5: $PASS PASSED, $FAIL FAILED ==="

TEST 1.6–1.7: SD-WAN Fabric Health

#!/bin/bash

## sdwan_fabric_health_test.sh


echo "=== TEST 1.6-1.7: SD-WAN FABRIC HEALTH ==="

VMANAGE="vmanage.abhavtech.com"
COOKIE="/tmp/cookies.txt"

## Login

curl -k -s -X POST https://$VMANAGE/j_security_check \
  -H "Content-Type: application/x-www-form-urlencoded" \
  -d "j_username=admin&j_password=<PASSWORD>" -c $COOKIE > /dev/null

echo ""
echo "--- TEST 1.6: Device Reachability ---"
DEVICES=$(curl -k -s -b $COOKIE "https://$VMANAGE/dataservice/device?type=vedge")
TOTAL=$(echo $DEVICES | python3 -c "import json,sys; d=json.load(sys.stdin); print(len(d.get('data', [])))")
REACHABLE=$(echo $DEVICES | python3 -c "import json,sys; d=json.load(sys.stdin); print(sum(1 for x in d['data'] if x.get('reachability')=='reachable'))")
echo "  Devices: $REACHABLE/$TOTAL reachable"
[[ "$REACHABLE" -eq 19 ]] && echo "  ✅ PASS: All 19 devices reachable" \
  || { echo "  ❌ FAIL: $((19 - REACHABLE)) devices unreachable"; \
       echo $DEVICES | python3 -c "import json,sys; d=json.load(sys.stdin); [print(f'  - {x[\"host-name\"]} ({x[\"system-ip\"]})') for x in d['data'] if x.get('reachability')!='reachable']"; }

echo ""
echo "--- TEST 1.7: BFD Sessions ---"
for HUB in mum-hub-01 lon-hub-01 nj-hub-01; do
  BFD_TOTAL=$(ssh admin@$HUB "show sdwan bfd sessions | wc -l" 2>/dev/null)
  BFD_UP=$(ssh admin@$HUB "show sdwan bfd sessions | grep Up | wc -l" 2>/dev/null)
  echo "  [$HUB]: $BFD_UP/$BFD_TOTAL BFD sessions UP"
  [[ "$BFD_UP" -ge 30 ]] && echo "  ✅ PASS: BFD health OK" || echo "  ❌ FAIL: Low BFD sessions"
done

echo ""
echo "--- TEST 1.8: OMP Route Distribution (VPN 1) ---"
for SITE in mum-hub-01 blr-isr-01 del-isr-01; do
  OMP_COUNT=$(ssh admin@$SITE "show sdwan omp routes vpn 1 | count" 2>/dev/null | tail -1)
  echo "  [$SITE]: $OMP_COUNT OMP routes in VPN 1"
  [[ "$OMP_COUNT" -ge 20 ]] && echo "  ✅ PASS" || echo "  ❌ FAIL: Low route count (expected 20+)"
done

TEST 1.9–1.10: Umbrella and ThousandEyes

## TEST 1.9: Umbrella Registration

echo "=== TEST 1.9: UMBRELLA DEVICE REGISTRATION ==="

for SITE in mum-hub-01 lon-hub-01 nj-hub-01 blr-isr-01 del-isr-01 hyd-br-01; do
  STATUS=$(ssh admin@$SITE "show umbrella config | grep -i status" 2>/dev/null)
  echo "$STATUS" | grep -qi "registered" && echo "  ✅ [$SITE]: Umbrella registered" \
    || echo "  ❌ [$SITE]: Umbrella NOT registered"
done

## TEST 1.10: ThousandEyes Agents

echo ""
echo "=== TEST 1.10: THOUSANDEYES AGENT STATUS ==="

## Use ThousandEyes API v7

TE_TOKEN="<TE_BEARER_TOKEN>"

curl -s -X GET "https://api.thousandeyes.com/v7/agents" \
  -H "Authorization: Bearer $TE_TOKEN" | \
  python3 -c "
import json, sys
data = json.load(sys.stdin)
agents = data.get('agents', [])
print(f'Total agents: {len(agents)}')
for a in agents:
    status = '✅' if a.get('enabled') and a.get('agentState') == 'Online' else '❌'
    print(f'{status} {a[\"agentName\"]:20} {a.get(\"agentState\",\"unknown\"):8} Last seen: {a.get(\"lastSeen\",\"N/A\")}')
"