Designing Private Network Connectivity for RAG-Capable Gen AI Apps on Google Cloud: What the March 2026 Reference Architecture Gets Right (and What You Still Need to Decide)

Enterprises love generative AI right up until the moment someone asks a simple question: “So… does any of this data touch the public internet?”

If your answer involves a long pause, a whiteboard marker, and the phrase “it depends,” you’re not alone. Retrieval-augmented generation (RAG) is a terrific way to make large language models more useful and less hallucinatory by grounding responses in your company’s own content. It also has a habit of dragging networking, identity, and data governance into the same room—where they immediately start arguing over who broke compliance.

On March 2, 2026, Google Cloud published a networking-focused piece—Designing private network connectivity for RAG-capable gen AI apps—by Ammett Williams, Developer Relations Engineer. The post points readers to a full reference architecture doc for “private connectivity for RAG-capable generative AI applications,” aimed at scenarios where system communications must use private IP addresses and “must not traverse the internet.”

This article expands that post into a practical, field-guide style walkthrough: what the architecture actually implies, which Google Cloud building blocks matter most, where teams trip up, and how to reason about trade-offs like Private Service Connect vs Private Google Access, or Network Connectivity Center vs a mess of peering links.

What “private connectivity” means for RAG (and what it doesn’t)

Google’s blog frames the goal clearly: keep communications private, using private IPs, without traffic traversing the public internet. That’s a networking requirement, not a magic compliance stamp. You still need to answer questions such as:

• Is “private” about transport (no public routing), about exposure (no public endpoints), about control (who can reach what), or all three?
• Do you need private input (data ingestion), private inter-service connectivity (service-to-service within cloud), private output (inference responses), or all of them?
• What counts as “internet” for your auditors—public IP space, public DNS names, or any traffic that isn’t pinned to private endpoints?

The reference architecture Google points to is primarily about building a system where data engineers can upload content privately, and users can query the RAG system privately, while the internal components communicate over private addressing and controlled paths. citeturn1view0turn0search4

A quick refresher: why RAG becomes a networking problem

RAG is conceptually simple: retrieve relevant context, add it to the prompt, generate a grounded response. Google’s post highlights the motivation—grounding improves relevance and can reduce hallucinations without retraining the model. citeturn1view0

But RAG is also a pipeline of separate subsystems, each with its own network and security needs:

• Data ingestion: engineers upload documents; the system chunks them; embeddings get generated.
• Vector storage / RAG datastore: stores embeddings and metadata; must be reachable from the serving layer.
• Serving: retrieves top-k context, builds prompts, calls the model endpoint, returns responses.
• Frontend: receives user requests and handles auth, rate limiting, and application logic.
• Model endpoint: hosted model APIs (e.g., Vertex AI) or your own deployed model service.

Any one of these can become a data exfiltration path if it has broad egress or public endpoints. When your input data is sensitive, the “default internet egress everywhere” model stops feeling modern and starts feeling like a dare.

The Google Cloud design pattern, in plain English

Google’s blog describes a regional design pattern with:

• An external network (on-prem or another cloud)
• A routing project
• A Shared VPC host project for the RAG VPC network
• Three service projects: data ingestion, serving, and frontend

It uses Cloud Interconnect or Cloud VPN for private connectivity from the external network into Google Cloud, Network Connectivity Center (NCC) to orchestrate routing between networks, Private Service Connect (PSC) to provide a private endpoint to Cloud Storage for ingestion, a load balancer plus Cloud Armor for the user-facing entry point, and VPC Service Controls (VPC-SC) to create a perimeter that reduces exfiltration risk. citeturn1view0turn0search4

Why so many projects?

Because “one project to rule them all” is how you end up with 200 people having Editor on the same place that stores your crown jewels. The architecture’s separate service projects align with typical enterprise controls:

• Blast-radius reduction: compromise in frontend doesn’t automatically imply write access to ingestion buckets or vector stores.
• IAM segmentation: different teams own different lifecycles and privileges.
• Billing and audit clarity: tracing spend and access patterns is easier when boundaries are explicit.

Shared VPC is Google Cloud’s standard way to centralize the network while keeping service ownership distributed across projects.

The two big flows: population and inference

The blog usefully separates network traffic into two flows: RAG population (ingestion) and inference (query/response). citeturn1view0

1) RAG population flow (ingestion)

In the reference, data engineers in the external network upload data over Cloud Interconnect or Cloud VPN into a routing VPC, then use a Private Service Connect endpoint to reach a Cloud Storage bucket without traversing the public internet. Data is processed in the ingestion subsystem; embeddings are generated via the model; vectors are written into the RAG datastore in the serving project. citeturn1view0turn0search4

This is a subtle but important detail: the “private upload” story isn’t just “use a VPN.” It’s “use a VPN/interconnect, then terminate into an internal endpoint that represents the managed service.” PSC is doing a lot of heavy lifting here.

2) Inference flow (user query)

User requests travel over Cloud Interconnect or Cloud VPN into the routing VPC, then to the RAG VPC via NCC. The request reaches an internal application load balancer protected by Cloud Armor, then to the frontend subsystem. Frontend calls serving; serving retrieves from the RAG datastore and calls the AI model; the response returns back along the same path. citeturn1view0turn0search4

The interesting point is what’s not in that story: there’s no public HTTP(S) endpoint required for the AI app. You can keep the entire customer interaction “inside” your private connectivity envelope—assuming your users can reach your private network through whatever hybrid connectivity you’ve designed.

Network Connectivity Center: the anti-spaghetti tool

NCC is described as the orchestration framework that manages connectivity between routing and RAG VPC networks using VPC spokes and hybrid spokes. citeturn1view0turn2search3

If you’ve ever tried to scale VPC peering across many projects, you know how quickly it turns into a graph that looks like modern art. NCC’s hub-and-spoke model is designed to reduce the operational complexity of pairwise peering, and it keeps inter-VPC traffic on Google’s network rather than the public internet. citeturn2search0

Hubs, spokes, and why your routing project matters

In NCC, a hub is the central resource; spokes represent attached networks or hybrid connectivity resources. NCC supports VPC spokes (for VPC-to-VPC), and hybrid spokes (HA VPN, Interconnect VLAN attachments, router appliances) for site-to-cloud and even site-to-site connectivity over Google’s backbone. citeturn2search3

The routing project in Google’s pattern is essentially your “landing zone” for external connectivity and controlled propagation of routes into the Shared VPC environment.

Design takeaway: keep route propagation intentional

The blog mentions Cloud Router and BGP route exchange, with NCC managing route propagation. citeturn1view0turn0search4 That’s crucial, because “private” architectures often fail not due to lack of encryption, but due to:

• Over-advertised routes (suddenly everything can reach everything)
• Under-advertised routes (nothing can reach the thing that matters at 2 a.m.)
• Confusing asymmetry (requests go in; responses take a different path and die)

NCC gives you a framework, but you still need strong network governance: route tables, segmentation, and a plan for how hybrid routes enter and leave your environment.

Private Service Connect: your “private endpoint” Swiss Army knife

PSC shows up twice in this space:

• PSC to Google APIs (consumer endpoints that let workloads reach Google APIs via internal IPs)
• PSC to managed services (private access to services like Cloud Storage endpoints or certain managed databases/services)

In the blog’s design, PSC provides a private endpoint in the routing VPC to reach Cloud Storage for ingestion. citeturn1view0turn0search4

PSC vs “just use Private Google Access”

Private Google Access (PGA) is commonly used to let VMs without external IPs reach Google APIs over Google’s network. PSC for Google APIs is a more explicit endpoint-based approach: you allocate internal IPs, wire DNS, and route to those endpoints. The Google Cloud docs call out practical considerations like required APIs (Compute Engine, Service Directory, Cloud DNS), firewall egress permitting traffic to the endpoint, and a key gotcha: PSC endpoints aren’t accessible from peered VPC networks. citeturn2search2

That last point is why NCC (or other connectivity approaches) matters: you need a plan for who can resolve and reach the endpoint, and across which network boundaries.

PSC for Vertex AI: making model access private

For RAG systems that call hosted models, the model endpoint can become your biggest “internet anxiety” trigger. Google provides guidance for setting up Private Service Connect interface (PSC-I) for Vertex AI resources, including recommended subnet sizing (Vertex AI recommends a /28), and IP range constraints for network attachments. citeturn0search0

There are also reference patterns for multicloud access—e.g., reaching the Vertex AI API from AWS through PSC for Google APIs combined with hybrid connectivity and DNS configuration. citeturn0search1

The point: if you want the “no public egress” story to hold up under scrutiny, you need to ensure model calls aren’t secretly leaving through default public endpoints.

VPC Service Controls: the perimeter that tries to stop “oops”

VPC Service Controls is Google’s answer to a very human problem: trusted identities doing untrusted things (accidentally or intentionally). It creates service perimeters around selected Google Cloud services (like Cloud Storage and BigQuery) to reduce data exfiltration risk. citeturn2search4

Google’s networking post includes VPC-SC as a core component, describing it as a managed security perimeter to mitigate data exfiltration risks. citeturn1view0

Why VPC-SC matters specifically for RAG

RAG frequently involves:

• Large document corpora in object storage
• Embedding pipelines that touch sensitive content
• Vector stores that may encode sensitive semantics
• Prompt logs, conversation histories, and evaluation datasets

Even if your transport is private, an engineer with broad permissions can still copy data to an unauthorized bucket or project. VPC-SC helps prevent access to resources outside the perimeter and restricts service operations that could move data out. citeturn2search4

It’s not a replacement for IAM least privilege—but it is a safety net, and in AI projects, safety nets are underrated until you need one.

Cloud Armor + Load Balancing: yes, even for “private” apps

The blog places Cloud Armor and an application load balancer in the frontend project to provide security and traffic management for user interaction. citeturn1view0

Some teams hear “private” and assume they can skip standard application protections because “only internal users can reach it.” Then those internal users bring their own devices, browsers, extensions, and occasionally malware. A private endpoint doesn’t exempt you from:

• WAF protections (SQLi, XSS patterns, protocol anomalies)
• DDoS resilience (internal systems can still be overwhelmed, and misconfigurations happen)
• Authentication and authorization layers

In fact, internal-only apps often get less testing and fewer guardrails—making them attractive targets.

Putting the pieces together: a practical checklist

Google’s blog post is short by design (it’s a blog), and it explicitly points you to the full architecture document for detailed IAM permissions and deployment considerations. citeturn1view0turn0search4

Here’s a pragmatic checklist to turn the pattern into an implementable plan:

1) Decide what must be private, with explicit scope

• Data ingestion paths (engineer-to-storage)
• User query paths (user-to-frontend)
• Service-to-service paths (frontend-to-serving, serving-to-vector store)
• Model API calls (serving-to-Vertex AI or to your own model endpoint)
• Management plane access (admins, CI/CD runners, workstations)

2) Pick the connectivity backbone

• Cloud Interconnect for predictable performance and dedicated connectivity (where appropriate)
• HA VPN for faster rollout or smaller-scale needs
• NCC hub-and-spoke to avoid peering sprawl and to manage route propagation at scale citeturn2search3turn2search0

3) Design DNS intentionally (don’t let it improvise)

Private endpoints are only useful if clients resolve the right names to the right internal IPs. PSC for Google APIs involves DNS and Service Directory, and Google’s docs even warn about conflicts if a private zone for p.googleapis.com already exists. citeturn2search2

In hybrid and multicloud scenarios, DNS becomes a first-class component of the architecture, not a footnote.

4) Use PSC where “private endpoint” semantics matter

• PSC to reach Cloud Storage ingestion endpoints privately (as in the reference) citeturn0search4
• PSC-I for Vertex AI to keep model calls private and control egress citeturn0search0

5) Add VPC Service Controls for exfiltration resistance

Especially if your RAG corpus includes regulated or sensitive information. VPC-SC is designed to mitigate data exfiltration risks from Google Cloud services by defining service perimeters and policy controls. citeturn2search4

6) Treat “frontend” like a production internet app (even if internal)

• Cloud Armor policies and logging
• Load balancing patterns that support blue/green or canary
• Rate limiting to avoid “accidental DDoS by enthusiastic employees”

Common pitfalls (learned the hard way, so you don’t have to)

Pitfall #1: Confusing private IPs with private data handling

Private routing reduces exposure, but you still need data classification, encryption at rest, key management, and tight IAM. “Private connectivity” is one layer, not the whole cake.

Pitfall #2: Letting egress stay open “for now”

RAG apps often start as prototypes. Prototypes love unrestricted egress because it makes everything work. Enterprises love restricting egress because it makes everything safe. You can guess who wins when the prototype becomes production on a Friday afternoon.

Decide early whether you’ll enforce egress restrictions, and how you’ll provide required access (PSC endpoints, approved NAT egress, allowlists, etc.).

Pitfall #3: Underestimating route and DNS complexity in multicloud

The multicloud reference architecture for accessing Vertex AI from AWS shows just how much glue is involved: VPN, BGP, custom route advertisements, and private hosted zones mapping Google API names to PSC IPs. citeturn0search1

Multicloud is not “plug in two clouds and enjoy.” It’s “become a part-time BGP operator.” Plan accordingly.

Pitfall #4: Treating vector stores like generic databases

Vector stores often sit on managed databases, search engines, or specialized services. Their network exposure matters because they represent derived knowledge from your corpus. Segment them like you would any sensitive datastore, and ensure only the serving subsystem can query them.

Industry context: why this architecture is showing up now

The date matters. Google’s blog post is dated March 2, 2026. citeturn1view0 Enterprises in 2025–2026 have largely moved past “can we do gen AI?” to “can we do gen AI without creating a new compliance incident category?”

Three forces are driving the push for private connectivity patterns:

• Regulatory scrutiny around data locality and access controls
• Board-level risk concerns about data leakage through AI systems
• Operational reality: many RAG corpora are built from internal documents never intended to leave private networks

In other words, the networking team is no longer “blocking innovation.” They’re preventing the AI team from accidentally inventing public document search for your M&A folder.

How to talk about this with stakeholders (without starting a war)

To security and compliance

Frame the architecture as layered controls:

• Private transport paths (Interconnect/VPN + NCC)
• Private endpoints to managed services (PSC)
• Perimeter-based exfiltration controls (VPC-SC)
• App-layer protection (Cloud Armor + LB)

This is easier to approve than “we promise the model won’t leak data.” Models don’t sign policies. Networks do.

To application teams

Emphasize that private connectivity is not merely a constraint—it’s an enabler for productionizing RAG with sensitive corpora. Once ingestion, retrieval, and inference can happen privately, more datasets become usable, and more business units say “yes.”

To networking teams

Give them a hub-and-spoke story they can operate. NCC’s intent is to reduce operational complexity compared to piles of peering, while keeping traffic within Google’s network. citeturn2search0

Comparisons: when this pattern makes sense (and when it’s overkill)

Great fit

• Heavily regulated industries (financial services, healthcare, public sector)
• Companies with significant on-prem presence and strict network segmentation
• RAG corpora containing confidential IP, customer data, or sensitive internal communications
• Organizations already operating Shared VPC and centralized network governance

Possibly overkill (for now)

• Early-stage prototypes with non-sensitive data
• Small teams without a networking operations function
• Workloads where internet-based access to managed services is acceptable and properly controlled

But beware the classic trap: “prototype today, production tomorrow.” If your roadmap includes sensitive corpora, it’s often cheaper to adopt private connectivity foundations early than to retrofit them later.

Internal linking opportunities for your own architecture docs

If you’re documenting this pattern internally (and you should), consider creating internal pages for:

• “RAG ingestion network pattern” (PSC to Cloud Storage, data engineer access, DNS)
• “Model API connectivity standards” (PSC-I for Vertex AI, endpoint naming, allowed regions)
• “NCC hub operations” (route propagation rules, spoke onboarding checklist, change management)
• “VPC-SC policy guide for AI projects” (perimeter design, ingress/egress rules, break-glass)
• “Frontend security baseline” (Cloud Armor policies, logging, rate limits)

This turns a one-time design into something your org can replicate safely.

What to read next (straight from Google, plus the original source)

The original RSS item points to Google Cloud’s blog post, which is the right starting point for understanding the reference architecture at a high level. The blog itself links to the full architecture document in Google Cloud’s Architecture Center, which contains significantly more detail. citeturn1view0turn0search4

Sources

• Google Cloud Blog: Designing private network connectivity for RAG-capable gen AI apps (Ammett Williams, March 2, 2026)
• Google Cloud Architecture Center: Private connectivity for RAG-capable generative AI applications
• Google Cloud Documentation: Network Connectivity Center overview
• Google Cloud Documentation: VPC spokes overview (Network Connectivity Center)
• Google Cloud Documentation: Access Google APIs through endpoints (Private Service Connect)
• Google Cloud Documentation: Overview of VPC Service Controls
• Google Cloud Documentation: Set up a Private Service Connect interface for Vertex AI resources
• Google Cloud Documentation: Allow multicloud access to protected resources from private endpoint outside a VPC Service Controls perimeter

Bas Dorland, Technology Journalist & Founder of dorland.org