Reference Architecture with Terraform: VM-Series in AWS, Centralized Design Model, Common NGFW Option

Palo Alto Networks produces several validated reference architecture design and deployment documentation guides, which describe well-architected and tested deployments. When deploying VM-Series in a public cloud, the reference architectures guide users toward the best security outcomes, whilst reducing rollout time and avoiding common integration efforts. The Terraform code presented here will deploy Palo Alto Networks VM-Series firewalls in AWS based on the centralized design; for a discussion of other options, please see the design guide from the reference architecture guides.

Reference Architecture Design

This code implements:

• a centralized design, which secures outbound, inbound, and east-west traffic flows using an AWS transit gateway (TGW). Application resources are segmented across multiple VPCs that connect in a hub-and-spoke topology, with a dedicated VPC for security services where the VM-Series are deployed

Detailed Architecture and Design

Centralized Design

This design supports interconnecting a large number of VPCs, with a scalable solution to secure outbound, inbound, and east-west traffic flows using a transit gateway to connect the VPCs. The centralized design model offers the benefits of a highly scalable design for multiple VPCs connecting to a central hub for inbound, outbound, and VPC-to-VPC traffic control and visibility. In the Centralized design model, you segment application resources across multiple VPCs that connect in a hub-and-spoke topology. The hub of the topology, or transit gateway, is the central point of connectivity between VPCs and Prisma Access or enterprise network resources attached through a VPN or AWS Direct Connect. This model has a dedicated VPC for security services where you deploy VM-Series firewalls for traffic inspection and control. The security VPC does not contain any application resources. The security VPC centralizes resources that multiple workloads can share. The TGW ensures that all spoke-to-spoke and spoke-to-enterprise traffic transits the VM-Series.

Auto Scaling VM-Series

Auto scaling: Public-cloud environments focus on scaling out a deployment instead of scaling up. This architectural difference stems primarily from the capability of public-cloud environments to dynamically increase or decrease the number of resources allocated to your environment. Using native AWS services like CloudWatch, auto scaling groups (ASG) and VM-Series automation features, the guide implements VM-Series that will scale in and out dynamically, as your protected workload demands fluctuate. The VM-Series firewalls are deployed in an auto scaling group, and are automatically registered to a Gateway Load Balancer. While bootstrapping the VM-Series, there are associations made automatically between VM-Series subinterfaces and the GWLB endpoints. Each VM-Series contains multiple network interfaces created by an AWS Lambda function.

Prerequisites

The following steps should be followed before deploying the Terraform code presented here.

1. Deploy Panorama e.g. by using Panorama example
2. Prepare device group, template, template stack in Panorama
3. Download and install plugin sw_fw_license for managing licenses
4. Configure bootstrap definition and license manager
5. Configure license API key
6. Configure security rules and NAT rules for outbound traffic
7. Configure interface management profile to enable health checks from GWLB
8. Configure network interfaces and subinterfaces, zones and virtual router in template

In example VM-Series are licensed using Panorama-Based Software Firewall License Management sw_fw_license, from which after configuring license manager values of panorama-server, auth-key, dgname, tplname can be used in terraform.tfvars file. Another way to bootstrap and license VM-Series is using VM Auth Key. This approach requires preparing license (auth code) in file stored in S3 bucket or putting it in authcodes option. More information can be found in document describing how to choose a bootstrap method. Please note, that other bootstrapping methods may requires additional changes in example code (e.g. adding options vm-auth-key, authcodes) and/or creating additional resources (e.g. S3 buckets).

Usage

1. Copy example.tfvars into terraform.tfvars
2. Review terraform.tfvars file, especially with lines commented by # TODO: update here
3. Initialize Terraform: terraform init
4. Prepare plan: terraform plan
5. Deploy infrastructure: terraform apply -auto-approve
6. Destroy infrastructure if needed: terraform destroy -auto-approve

Additional Reading

Lambda function

Lambda function is used to handle correct lifecycle action:

• instance launch or
• instance terminate

In case of creating VM-Series, there are performed below actions, which cannot be achieved in AWS launch template:

• change setting source_dest_check for first network interface (data plane)
• setup additional network interfaces (with optional possibility to attach EIP)

In case of destroying VM-Series, there is performed below action:

• clean EIP

Moreover having Lambda function executed while scaling out or in gives more options for extension e.g. delicesning VM-Series just after terminating instance.

Autoscaling

AWS Auto Scaling monitors VM-Series and automatically adjusts capacity to maintain steady, predictable performance at the lowest possible cost. For autoscaling there are 10 metrics available from vmseries plugin:

• DataPlaneCPUUtilizationPct
• DataPlanePacketBufferUtilization
• panGPGatewayUtilizationPct
• panGPGWUtilizationActiveTunnels
• panSessionActive
• panSessionConnectionsPerSecond
• panSessionSslProxyUtilization
• panSessionThroughputKbps
• panSessionThroughputPps
• panSessionUtilization

Using that metrics there can be configured different scaling plans. Below there are some examples, which can be used. All examples are based on target tracking configuration in scaling plan. Below code is already embedded into asg module:

  scaling_instruction {
    max_capacity       = var.max_size
    min_capacity       = var.min_size
    resource_id        = format("autoScalingGroup/%s", aws_autoscaling_group.this.name)
    scalable_dimension = "autoscaling:autoScalingGroup:DesiredCapacity"
    service_namespace  = "autoscaling"
    target_tracking_configuration {
      customized_scaling_metric_specification {
        metric_name = var.scaling_metric_name
        namespace   = var.scaling_cloudwatch_namespace
        statistic   = var.scaling_statistic
      }
      target_value = var.scaling_target_value
    }
  }

Using metrics from vmseries plugin we can defined multiple scaling configurations e.g.:

• based on number of active sessions:

metric_name  = "panSessionActive"
target_value = 75
statistic    = "Average"

• based on data plane CPU utilization and average value above 75%:

metric_name  = "DataPlaneCPUUtilizationPct"
target_value = 75
statistic    = "Average"

• based on data plane packet buffer utilization and max value above 80%

metric_name  = "DataPlanePacketBufferUtilization"
target_value = 80
statistic    = "Maximum"

Reference

Requirements

Name	Version
terraform	>= 1.0.0, < 2.0.0
aws	~> 5.17
local	~> 2.4.0

Providers

Name	Version
aws	~> 5.17

Modules

Name	Source	Version
app_lb	../../modules/nlb	n/a
gwlb	../../modules/gwlb	n/a
gwlbe_endpoint	../../modules/gwlb_endpoint_set	n/a
natgw_set	../../modules/nat_gateway_set	n/a
public_alb	../../modules/alb	n/a
public_nlb	../../modules/nlb	n/a
subnet_sets	../../modules/subnet_set	n/a
transit_gateway	../../modules/transit_gateway	n/a
transit_gateway_attachment	../../modules/transit_gateway_attachment	n/a
vm_series_asg	../../modules/asg	n/a
vpc	../../modules/vpc	n/a
vpc_routes	../../modules/vpc_route	n/a

Resources

Name	Type
aws_ec2_transit_gateway_route.from_security_to_panorama	resource
aws_ec2_transit_gateway_route.from_spokes_to_security	resource
aws_iam_instance_profile.spoke_vm_iam_instance_profile	resource
aws_iam_instance_profile.vm_series_iam_instance_profile	resource
aws_iam_role.spoke_vm_ec2_iam_role	resource
aws_iam_role.vm_series_ec2_iam_role	resource
aws_iam_role_policy.vm_series_ec2_iam_policy	resource
aws_instance.spoke_vms	resource
aws_ami.this	data source
aws_caller_identity.this	data source
aws_ebs_default_kms_key.current	data source
aws_kms_alias.current_arn	data source
aws_partition.this	data source

Inputs

Name	Description	Type	Default	Required
global_tags	Global tags configured for all provisioned resources	any	n/a	yes
gwlb_endpoints	A map defining GWLB endpoints. Following properties are available: - name: name of the GWLB endpoint - gwlb: key of GWLB - vpc: key of VPC - vpc_subnet: key of the VPC and subnet connected by '-' character - act_as_next_hop: set to true if endpoint is part of an IGW route table e.g. for inbound traffic - to_vpc_subnets: subnets to which traffic from IGW is routed to the GWLB endpoint Example: gwlb_endpoints = { security_gwlb_eastwest = { name = "eastwest-gwlb-endpoint" gwlb = "security_gwlb" vpc = "security_vpc" vpc_subnet = "security_vpc-gwlbe_eastwest" act_as_next_hop = false to_vpc_subnets = null } }	map(object({ name = string gwlb = string vpc = string vpc_subnet = string act_as_next_hop = bool to_vpc_subnets = string }))	{}	no
gwlbs	A map defining Gateway Load Balancers. Following properties are available: - name: name of the GWLB - vpc_subnet: key of the VPC and subnet connected by '-' character Example: gwlbs = { security_gwlb = { name = "security-gwlb" vpc_subnet = "security_vpc-gwlb" } }	map(object({ name = string vpc_subnet = string }))	{}	no
name_prefix	Prefix used in names for the resources (VPCs, EC2 instances, autoscaling groups etc.)	string	n/a	yes
natgws	A map defining NAT Gateways. Following properties are available: - name: name of NAT Gateway - vpc_subnet: key of the VPC and subnet connected by '-' character Example: natgws = { security_nat_gw = { name = "natgw" vpc_subnet = "security_vpc-natgw" } }	map(object({ name = string vpc_subnet = string }))	{}	no
panorama_attachment	A object defining TGW attachment and CIDR for Panorama. Following properties are available: - transit_gateway_attachment_id: ID of attachment for Panorama - vpc_cidr: CIDR of the VPC, where Panorama is deployed Example: panorama = { transit_gateway_attachment_id = "tgw-attach-123456789" vpc_cidr = "10.255.0.0/24" }	object({ transit_gateway_attachment_id = string vpc_cidr = string })	null	no
region	AWS region used to deploy whole infrastructure	string	n/a	yes
spoke_lbs	A map defining Network Load Balancers deployed in spoke VPCs. Following properties are available: - vpc_subnet: key of the VPC and subnet connected by '-' character - vms: keys of spoke VMs Example: spoke_lbs = { "app1-nlb" = { vpc_subnet = "app1_vpc-app1_lb" vms = ["app1_vm01", "app1_vm02"] } }	map(object({ vpc_subnet = string vms = list(string) }))	{}	no
spoke_vms	A map defining VMs in spoke VPCs. Following properties are available: - az: name of the Availability Zone - vpc: name of the VPC (needs to be one of the keys in map vpcs) - vpc_subnet: key of the VPC and subnet connected by '-' character - security_group: security group assigned to ENI used by VM - type: EC2 type VM Example: spoke_vms = { "app1_vm01" = { az = "eu-central-1a" vpc = "app1_vpc" vpc_subnet = "app1_vpc-app1_vm" security_group = "app1_vm" type = "t2.micro" } }	map(object({ az = string vpc = string vpc_subnet = string security_group = string type = string }))	{}	no
ssh_key_name	Name of the SSH key pair existing in AWS key pairs and used to authenticate to VM-Series or test boxes	string	n/a	yes
tgw	A object defining Transit Gateway. Following properties are available: - create: set to false, if existing TGW needs to be reused - id: id of existing TGW or null - name: name of TGW to create or use - asn: ASN number - route_tables: map of route tables - attachments: map of TGW attachments Example: tgw = { create = true id = null name = "tgw" asn = "64512" route_tables = { "from_security_vpc" = { create = true name = "from_security" } } attachments = { security = { name = "vmseries" vpc_subnet = "security_vpc-tgw_attach" route_table = "from_security_vpc" propagate_routes_to = "from_spoke_vpc" } } }	object({ create = bool id = string name = string asn = string route_tables = map(object({ create = bool name = string })) attachments = map(object({ name = string vpc_subnet = string route_table = string propagate_routes_to = string })) })	null	no
vmseries_asgs	A map defining Autoscaling Groups with VM-Series instances. Following properties are available: - bootstrap_options: VM-Seriess bootstrap options used to connect to Panorama - panos_version: PAN-OS version used for VM-Series - ebs_kms_id: alias for AWS KMS used for EBS encryption in VM-Series - vpc: key of VPC - gwlb: key of GWLB - interfaces: configuration of network interfaces for VM-Series used by Lamdba while provisioning new VM-Series in autoscaling group - subinterfaces: configuration of network subinterfaces used to map with GWLB endpoints - asg: the number of Amazon EC2 instances that should be running in the group (desired, minimum, maximum) - scaling_plan: scaling plan with attributes - enabled: true if automatic dynamic scaling policy should be created - metric_name: name of the metric used in dynamic scaling policy - target_value: target value for the metric used in dynamic scaling policy - statistic: statistic of the metric. Valid values: Average, Maximum, Minimum, SampleCount, Sum - cloudwatch_namespace: name of CloudWatch namespace, where metrics are available (it should be the same as namespace configured in VM-Series plugin in PAN-OS) - tags: tags configured for dynamic scaling policy Example: vmseries_asgs = { main_asg = { bootstrap_options = { mgmt-interface-swap = "enable" plugin-op-commands = "panorama-licensing-mode-on,aws-gwlb-inspect:enable,aws-gwlb-overlay-routing:enable" # TODO: update here panorama-server = "" # TODO: update here auth-key = "" # TODO: update here dgname = "" # TODO: update here tplname = "" # TODO: update here dhcp-send-hostname = "yes" # TODO: update here dhcp-send-client-id = "yes" # TODO: update here dhcp-accept-server-hostname = "yes" # TODO: update here dhcp-accept-server-domain = "yes" # TODO: update here } panos_version = "10.2.3" # TODO: update here ebs_kms_id = "alias/aws/ebs" # TODO: update here vpc = "security_vpc" gwlb = "security_gwlb" interfaces = { private = { device_index = 0 security_group = "vmseries_private" subnet = { "privatea" = "eu-central-1a", "privateb" = "eu-central-1b" } create_public_ip = false source_dest_check = false } mgmt = { device_index = 1 security_group = "vmseries_mgmt" subnet = { "mgmta" = "eu-central-1a", "mgmtb" = "eu-central-1b" } create_public_ip = true source_dest_check = true } public = { device_index = 2 security_group = "vmseries_public" subnet = { "publica" = "eu-central-1a", "publicb" = "eu-central-1b" } create_public_ip = false source_dest_check = false } } subinterfaces = { inbound = { app1 = { gwlb_endpoint = "app1_inbound" subinterface = "ethernet1/1.11" } app2 = { gwlb_endpoint = "app2_inbound" subinterface = "ethernet1/1.12" } } outbound = { only_1_outbound = { gwlb_endpoint = "security_gwlb_outbound" subinterface = "ethernet1/1.20" } } eastwest = { only_1_eastwest = { gwlb_endpoint = "security_gwlb_eastwest" subinterface = "ethernet1/1.30" } } } asg = { desired_cap = 0 min_size = 0 max_size = 4 lambda_execute_pip_install_once = true } scaling_plan = { enabled = true # TODO: update here metric_name = "panSessionActive" # TODO: update here target_value = 75 # TODO: update here statistic = "Average" # TODO: update here cloudwatch_namespace = "example-vmseries" # TODO: update here tags = { ManagedBy = "terraform" } } application_lb = null network_lb = null } }	map(object({ bootstrap_options = object({ mgmt-interface-swap = string plugin-op-commands = string panorama-server = string auth-key = string dgname = string tplname = string dhcp-send-hostname = string dhcp-send-client-id = string dhcp-accept-server-hostname = string dhcp-accept-server-domain = string }) panos_version = string ebs_kms_id = string vpc = string gwlb = string interfaces = map(object({ device_index = number security_group = string subnet = map(string) create_public_ip = bool source_dest_check = bool })) subinterfaces = map(map(object({ gwlb_endpoint = string subinterface = string }))) asg = object({ desired_cap = number min_size = number max_size = number lambda_execute_pip_install_once = bool }) scaling_plan = object({ enabled = bool metric_name = string target_value = number statistic = string cloudwatch_namespace = string tags = map(string) }) application_lb = object({ name = string rules = any }) network_lb = object({ name = string rules = any }) }))	{}	no
vpcs	A map defining VPCs with security groups and subnets. Following properties are available: - name: VPC name - cidr: CIDR for VPC - security_groups: map of security groups - subnets: map of subnets with properties: - az: availability zone - set: internal identifier referenced by main.tf - routes: map of routes with properties: - vpc_subnet - built from key of VPCs concatenate with - and key of subnet in format: VPCKEY-SUBNETKEY - next_hop_key - must match keys use to create TGW attachment, IGW, GWLB endpoint or other resources - next_hop_type - internet_gateway, nat_gateway, transit_gateway_attachment or gwlbe_endpoint Example: vpcs = { example_vpc = { name = "example-spoke-vpc" cidr = "10.104.0.0/16" nacls = { trusted_path_monitoring = { name = "trusted-path-monitoring" rules = { allow_inbound = { rule_number = 300 egress = false protocol = "-1" rule_action = "allow" cidr_block = "0.0.0.0/0" from_port = null to_port = null } } } } security_groups = { example_vm = { name = "example_vm" rules = { all_outbound = { description = "Permit All traffic outbound" type = "egress", from_port = "0", to_port = "0", protocol = "-1" cidr_blocks = ["0.0.0.0/0"] } } } } subnets = { "10.104.0.0/24" = { az = "eu-central-1a", set = "vm", nacl = null } "10.104.128.0/24" = { az = "eu-central-1b", set = "vm", nacl = null } } routes = { vm_default = { vpc_subnet = "app1_vpc-app1_vm" to_cidr = "0.0.0.0/0" next_hop_key = "app1" next_hop_type = "transit_gateway_attachment" } } } }	map(object({ name = string cidr = string nacls = map(object({ name = string rules = map(object({ rule_number = number egress = bool protocol = string rule_action = string cidr_block = string from_port = string to_port = string })) })) security_groups = any subnets = map(object({ az = string set = string nacl = string })) routes = map(object({ vpc_subnet = string to_cidr = string next_hop_key = string next_hop_type = string })) }))	{}	no

Outputs

Name	Description
app_inspected_dns_name	FQDN of App Internal Load Balancer. Can be used in VM-Series configuration to balance traffic between the application instances.
public_alb_dns_name	FQDN of VM-Series External Application Load Balancer used in centralized design.
public_nlb_dns_name	FQDN of VM-Series External Network Load Balancer used in centralized design.