Reference Architecture with Terraform: VM-Series in AWS, Combined Design Model, Common NGFW Option with Autoscaling

Palo Alto Networks produces several validated reference architecture design and deployment documentation guides, which describe well-architected and tested deployments. When deploying VM-Series in a public cloud, the reference architectures guide users toward the best security outcomes, whilst reducing rollout time and avoiding common integration efforts. The Terraform code presented here will deploy Palo Alto Networks VM-Series firewalls in AWS based on the centralized design; for a discussion of other options, please see the design guide from the reference architecture guides.

Reference Architecture Design

This code implements:

• a centralized design, which secures outbound, inbound, and east-west traffic flows using an AWS transit gateway (TGW). Application resources are segmented across multiple VPCs that connect in a hub-and-spoke topology, with a dedicated VPC for security services where the VM-Series are deployed
• a combined model for inbound traffic, where an AWS gateway load balancer (GWLB) is used to forward inbound traffic to the VM-Series in the security services VPC, as well as outbound and east-west traffic
• auto scaling for the VM-Series, where an AWS auto scaling group (ASG) is used to provision VM-Series that will scale in and out dynamically, as workload demands fluctuate

Detailed Architecture and Design

Centralized Design

This design supports interconnecting a large number of VPCs, with a scalable solution to secure outbound, inbound, and east-west traffic flows using a transit gateway to connect the VPCs. The centralized design model offers the benefits of a highly scalable design for multiple VPCs connecting to a central hub for inbound, outbound, and VPC-to-VPC traffic control and visibility. In the Centralized design model, you segment application resources across multiple VPCs that connect in a hub-and-spoke topology. The hub of the topology, or transit gateway, is the central point of connectivity between VPCs and Prisma Access or enterprise network resources attached through a VPN or AWS Direct Connect. This model has a dedicated VPC for security services where you deploy VM-Series firewalls for traffic inspection and control. The security VPC does not contain any application resources. The security VPC centralizes resources that multiple workloads can share. The TGW ensures that all spoke-to-spoke and spoke-to-enterprise traffic transits the VM-Series.

Combined Model for Inbound Traffic

Inbound traffic originates outside your VPCs and is destined to applications or services hosted within your VPCs, such as web or application servers. The combined model implements inbound security by using the VM-Series and Gateway Load Balancer (GWLB) in a Security VPC, with distributed GWLB endpoints in the application VPCs. Unlike with outbound traffic, this design option does not use the transit gateway for traffic forwarding between the security VPC and the application VPCs.

Auto Scaling VM-Series

Auto scaling: Public-cloud environments focus on scaling out a deployment instead of scaling up. This architectural difference stems primarily from the capability of public-cloud environments to dynamically increase or decrease the number of resources allocated to your environment. Using native AWS services like CloudWatch, auto scaling groups (ASG) and VM-Series automation features, the guide implements VM-Series that will scale in and out dynamically, as your protected workload demands fluctuate. The VM-Series firewalls are deployed in an auto scaling group, and are automatically registered to a Gateway Load Balancer. While bootstrapping the VM-Series, there are associations made automatically between VM-Series subinterfaces and the GWLB endpoints. Each VM-Series contains multiple network interfaces created by an AWS Lambda function.

Prerequisites

The following steps should be followed before deploying the Terraform code presented here.

1. Deploy Panorama e.g. by using Panorama example
2. Prepare device group, template, template stack in Panorama
3. Download and install plugin sw_fw_license for managing licenses
4. Configure bootstrap definition and license manager
5. Configure license API key
6. Configure security rules and NAT rules for outbound traffic
7. Configure interface management profile to enable health checks from GWLB
8. Configure network interfaces and subinterfaces, zones and virtual router in template
9. Configure static routes with path monitoring
10. Configure VPC peering between VPC with Panorama and VPC with VM-Series in autoscaling group (after deploying that example)

Details - static routes with path monitoring

Using multiple template stacks, one for each AZ complicates autoscaling and the Panorama Licensing plugin configuration. The virtual router (VR) configuration combined with path monitoring outlined below avoids using AZ-specific template stacks and variables.

Virtual Router Configuration

1. Create static routes for all internally routed CIDRs
2. Set the next hop to the default gateway IP of the trust subnet of the corresponding availability zone, which the firewall is connected to.
3. Set a unique metric value per AZ so that it doesn't overlap with other routes with the same destinations.
4. Enable Path Monitoring for the route.

• Source IP: DHCP
• Destination IP: Next Hop IP of the subnet of the corresponding AZ.

The AWS NACL applied to the trust subnets blocks the path monitor from pinging default gateways of the trust subnets in the other availability zones. This will cause the firewall to remove all routes that don't apply to the Availability zone it is in.

Below there is shown example of VR configuration with static routes and path monitoring:

Name	Destination	Next Hop	Metric	Path Monitor Destination IP
app1_az1	10.104.0.0/16	10.100.1.1	11	10.100.1.1
app2_az1	10.105.0.0/16	10.100.1.1	11	10.100.1.1
app1_az2	10.104.0.0/16	10.100.65.1	12	10.100.65.1
app2_az2	10.105.0.0/16	10.100.65.1	12	10.100.65.1
health_az1	10.100.0.0/16	10.100.1.1	11	10.100.1.1
health_az2	10.100.0.0/16	10.100.65.1	12	10.100.65.1

An example XML configuration snippet (for PANOS 10.2.3) of the described configuration can be found here, which after importing to Panorama, can be merged using the command:

load config partial mode merge from-xpath /config/devices/entry/template/entry[@name='asg'] to-xpath /config/devices/entry/template/entry[@name='asg'] from template-asg-path-monitoring.xml

Usage

NAT Gateway Option

There are two use cases supported in this example. You can select your preferred use case by using the applicable tfvars file for your use case.

• example-natgw-lambda-vpc.tfvars - with NAT Gateway presented in topology, where NAT Gateway is used for Lambda working in VPC for autoscaling group and for VM-Series instances, which for untrust interfaces don't have public IP
• example-no-natgw-lambda-no-vpc.tfvars - without NAT Gateway, where Lambda is not working in VPC and each VM-Series instance in autoscaling group has untrust interface with public IP

VM-Series delicensing

After scale in event, VM-Series needs to be delicensed by sw_fw_license plugin in Panorama. There are 2 possible approaches:

• enable option for plugin sw_fw_license to deactive firewall after being disconnected for N hours, where 1 <= N <= 24 hours
• use event-based approach and do delicense in Lambda in Python code, just after scale in, by executing command request plugins sw_fw_license deactivate license-manager LICENSE_MANAGER_NAME devices member VM_SERIES_SERIAL_NUMBER

Module asg is supporting both approaches. In example-natgw-lambda-vpc.tfvars Lambda is configured to be deployed in VPC and do delicense in Lambda in Python code. In example-no-natgw-lambda-no-vpc.tfvars Lambda is configured to be deployed outside VPC, without connection to Panorama and without executing any command on plugin sw_fw_license.

If event-based approach is being used, then additional prerequisites - configuration of connection with both Panoramas:

• go to AWS Systems Manager -> Parameter Store
• create new parameter with type SecureString and data:

{
    "username": "ACCOUNT",
    "password": "PASSWORD",
    "panorama1": "IP_ADDRESS",
    "panorama2": "IP_ADDRESS",
    "license_manager": "LICENSE_MANAGER_NAME"
}

• name of the parameter needs to be used in terraform.tfvars e.g.

    delicense = {
      enabled        = true
      ssm_param_name = "NAME_OF_THE_SECURE_STRING_PARAMETER"
    }

Deployment Steps

1. Copy example-no-natgw-lambda-no-vpc.tfvars or example-natgw-lambda-vpc.tfvars into terraform.tfvars
2. Review terraform.tfvars file, especially with lines commented by # TODO: update here
3. Initialize Terraform: terraform init
4. Prepare plan: terraform plan
5. Deploy infrastructure: terraform apply -auto-approve
6. Destroy infrastructure if needed: terraform destroy -auto-approve

Additional Reading

Lambda function

Lambda function is used to handle correct lifecycle action:

• instance launch or
• instance terminate

In case of creating VM-Series, there are performed below actions, which cannot be achieved in AWS launch template:

• change setting source_dest_check for first network interface (data plane)
• setup additional network interfaces (with optional possibility to attach EIP)

In case of destroying VM-Series, there is performed below action:

• clean EIP

Moreover having Lambda function executed while scaling out or in gives more options for extension e.g. delicesning VM-Series just after terminating instance.

Autoscaling

AWS Auto Scaling monitors VM-Series and automatically adjusts capacity to maintain steady, predictable performance at the lowest possible cost. For autoscaling there are 10 metrics available from vmseries plugin:

• DataPlaneCPUUtilizationPct
• DataPlanePacketBufferUtilization
• panGPGatewayUtilizationPct
• panGPGWUtilizationActiveTunnels
• panSessionActive
• panSessionConnectionsPerSecond
• panSessionSslProxyUtilization
• panSessionThroughputKbps
• panSessionThroughputPps
• panSessionUtilization

Using that metrics there can be configured different scaling plans. Below there are some examples, which can be used. All examples are based on target tracking configuration in scaling plan. Below code is already embedded into asg module:

  scaling_instruction {
    max_capacity       = var.max_size
    min_capacity       = var.min_size
    resource_id        = format("autoScalingGroup/%s", aws_autoscaling_group.this.name)
    scalable_dimension = "autoscaling:autoScalingGroup:DesiredCapacity"
    service_namespace  = "autoscaling"
    target_tracking_configuration {
      customized_scaling_metric_specification {
        metric_name = var.scaling_metric_name
        namespace   = var.scaling_cloudwatch_namespace
        statistic   = var.scaling_statistic
      }
      target_value = var.scaling_target_value
    }
  }

Using metrics from vmseries plugin we can defined multiple scaling configurations e.g.:

• based on number of active sessions:

metric_name  = "panSessionActive"
target_value = 75
statistic    = "Average"

• based on data plane CPU utilization and average value above 75%:

metric_name  = "DataPlaneCPUUtilizationPct"
target_value = 75
statistic    = "Average"

• based on data plane packet buffer utilization and max value above 80%

metric_name  = "DataPlanePacketBufferUtilization"
target_value = 80
statistic    = "Maximum"

Reference

Requirements

Name	Version
terraform	>= 1.0.0, < 2.0.0
aws	~> 5.17

Providers

Name	Version
aws	~> 5.17

Modules

Name	Source	Version
app_lb	../../modules/nlb	n/a
gwlb	../../modules/gwlb	n/a
gwlbe_endpoint	../../modules/gwlb_endpoint_set	n/a
natgw_set	../../modules/nat_gateway_set	n/a
subnet_sets	../../modules/subnet_set	n/a
transit_gateway	../../modules/transit_gateway	n/a
transit_gateway_attachment	../../modules/transit_gateway_attachment	n/a
vm_series_asg	../../modules/asg	n/a
vpc	../../modules/vpc	n/a
vpc_routes	../../modules/vpc_route	n/a

Resources

Name	Type
aws_ec2_transit_gateway_route.from_security_to_panorama	resource
aws_ec2_transit_gateway_route.from_spokes_to_security	resource
aws_iam_instance_profile.spoke_vm_iam_instance_profile	resource
aws_iam_instance_profile.vm_series_iam_instance_profile	resource
aws_iam_role.spoke_vm_ec2_iam_role	resource
aws_iam_role.vm_series_ec2_iam_role	resource
aws_iam_role_policy.vm_series_ec2_iam_policy	resource
aws_instance.spoke_vms	resource
aws_ami.this	data source
aws_caller_identity.this	data source
aws_ebs_default_kms_key.current	data source
aws_kms_alias.current_arn	data source
aws_partition.this	data source

Inputs

Name	Description	Type	Default	Required
global_tags	Global tags configured for all provisioned resources	any	n/a	yes
gwlb_endpoints	A map defining GWLB endpoints. Following properties are available: - name: name of the GWLB endpoint - gwlb: key of GWLB - vpc: key of VPC - vpc_subnet: key of the VPC and subnet connected by '-' character - act_as_next_hop: set to true if endpoint is part of an IGW route table e.g. for inbound traffic - to_vpc_subnets: subnets to which traffic from IGW is routed to the GWLB endpoint Example: gwlb_endpoints = { security_gwlb_eastwest = { name = "eastwest-gwlb-endpoint" gwlb = "security_gwlb" vpc = "security_vpc" vpc_subnet = "security_vpc-gwlbe_eastwest" act_as_next_hop = false to_vpc_subnets = null } }	map(object({ name = string gwlb = string vpc = string vpc_subnet = string act_as_next_hop = bool to_vpc_subnets = string }))	{}	no
gwlbs	A map defining Gateway Load Balancers. Following properties are available: - name: name of the GWLB - vpc_subnet: key of the VPC and subnet connected by '-' character Example: gwlbs = { security_gwlb = { name = "security-gwlb" vpc_subnet = "security_vpc-gwlb" } }	map(object({ name = string vpc_subnet = string }))	{}	no
name_prefix	Prefix used in names for the resources (VPCs, EC2 instances, autoscaling groups etc.)	string	n/a	yes
natgws	A map defining NAT Gateways. Following properties are available: - name: name of NAT Gateway - vpc_subnet: key of the VPC and subnet connected by '-' character Example: natgws = { security_nat_gw = { name = "natgw" vpc_subnet = "security_vpc-natgw" } }	map(object({ name = string vpc_subnet = string }))	{}	no
panorama_attachment	A object defining TGW attachment and CIDR for Panorama. Following properties are available: - transit_gateway_attachment_id: ID of attachment for Panorama - vpc_cidr: CIDR of the VPC, where Panorama is deployed Example: panorama = { transit_gateway_attachment_id = "tgw-attach-123456789" vpc_cidr = "10.255.0.0/24" }	object({ transit_gateway_attachment_id = string vpc_cidr = string })	null	no
region	AWS region used to deploy whole infrastructure	string	n/a	yes
spoke_lbs	A map defining Network Load Balancers deployed in spoke VPCs. Following properties are available: - vpc_subnet: key of the VPC and subnet connected by '-' character - vms: keys of spoke VMs Example: spoke_lbs = { "app1-nlb" = { vpc_subnet = "app1_vpc-app1_lb" vms = ["app1_vm01", "app1_vm02"] } }	map(object({ vpc_subnet = string vms = list(string) }))	{}	no
spoke_vms	A map defining VMs in spoke VPCs. Following properties are available: - az: name of the Availability Zone - vpc: name of the VPC (needs to be one of the keys in map vpcs) - vpc_subnet: key of the VPC and subnet connected by '-' character - security_group: security group assigned to ENI used by VM - type: EC2 type VM Example: spoke_vms = { "app1_vm01" = { az = "eu-central-1a" vpc = "app1_vpc" vpc_subnet = "app1_vpc-app1_vm" security_group = "app1_vm" type = "t2.micro" } }	map(object({ az = string vpc = string vpc_subnet = string security_group = string type = string }))	{}	no
ssh_key_name	Name of the SSH key pair existing in AWS key pairs and used to authenticate to VM-Series or test boxes	string	n/a	yes
tgw	A object defining Transit Gateway. Following properties are available: - create: set to false, if existing TGW needs to be reused - id: id of existing TGW or null - name: name of TGW to create or use - asn: ASN number - route_tables: map of route tables - attachments: map of TGW attachments Example: tgw = { create = true id = null name = "tgw" asn = "64512" route_tables = { "from_security_vpc" = { create = true name = "from_security" } } attachments = { security = { name = "vmseries" vpc_subnet = "security_vpc-tgw_attach" route_table = "from_security_vpc" propagate_routes_to = "from_spoke_vpc" } } }	object({ create = bool id = string name = string asn = string route_tables = map(object({ create = bool name = string })) attachments = map(object({ name = string vpc_subnet = string route_table = string propagate_routes_to = string })) })	null	no
vmseries_asgs	A map defining Autoscaling Groups with VM-Series instances. Following properties are available: - bootstrap_options: VM-Seriess bootstrap options used to connect to Panorama - panos_version: PAN-OS version used for VM-Series - ebs_kms_id: alias for AWS KMS used for EBS encryption in VM-Series - vpc: key of VPC - gwlb: key of GWLB - interfaces: configuration of network interfaces for VM-Series used by Lamdba while provisioning new VM-Series in autoscaling group - subinterfaces: configuration of network subinterfaces used to map with GWLB endpoints - asg: the number of Amazon EC2 instances that should be running in the group (desired, minimum, maximum) - scaling_plan: scaling plan with attributes - enabled: true if automatic dynamic scaling policy should be created - metric_name: name of the metric used in dynamic scaling policy - target_value: target value for the metric used in dynamic scaling policy - statistic: statistic of the metric. Valid values: Average, Maximum, Minimum, SampleCount, Sum - cloudwatch_namespace: name of CloudWatch namespace, where metrics are available (it should be the same as namespace configured in VM-Series plugin in PAN-OS) - tags: tags configured for dynamic scaling policy Example: vmseries_asgs = { main_asg = { bootstrap_options = { mgmt-interface-swap = "enable" plugin-op-commands = "panorama-licensing-mode-on,aws-gwlb-inspect:enable,aws-gwlb-overlay-routing:enable" # TODO: update here panorama-server = "" # TODO: update here auth-key = "" # TODO: update here dgname = "" # TODO: update here tplname = "" # TODO: update here dhcp-send-hostname = "yes" # TODO: update here dhcp-send-client-id = "yes" # TODO: update here dhcp-accept-server-hostname = "yes" # TODO: update here dhcp-accept-server-domain = "yes" # TODO: update here } panos_version = "10.2.3" # TODO: update here ebs_kms_id = "alias/aws/ebs" # TODO: update here vpc = "security_vpc" gwlb = "security_gwlb" interfaces = { private = { device_index = 0 security_group = "vmseries_private" subnet = { "privatea" = "eu-central-1a", "privateb" = "eu-central-1b" } create_public_ip = false source_dest_check = false } mgmt = { device_index = 1 security_group = "vmseries_mgmt" subnet = { "mgmta" = "eu-central-1a", "mgmtb" = "eu-central-1b" } create_public_ip = true source_dest_check = true } public = { device_index = 2 security_group = "vmseries_public" subnet = { "publica" = "eu-central-1a", "publicb" = "eu-central-1b" } create_public_ip = false source_dest_check = false } } subinterfaces = { inbound = { app1 = { gwlb_endpoint = "app1_inbound" subinterface = "ethernet1/1.11" } app2 = { gwlb_endpoint = "app2_inbound" subinterface = "ethernet1/1.12" } } outbound = { only_1_outbound = { gwlb_endpoint = "security_gwlb_outbound" subinterface = "ethernet1/1.20" } } eastwest = { only_1_eastwest = { gwlb_endpoint = "security_gwlb_eastwest" subinterface = "ethernet1/1.30" } } } asg = { desired_cap = 0 min_size = 0 max_size = 4 lambda_execute_pip_install_once = true } scaling_plan = { enabled = true # TODO: update here metric_name = "panSessionActive" # TODO: update here target_value = 75 # TODO: update here statistic = "Average" # TODO: update here cloudwatch_namespace = "example-vmseries" # TODO: update here tags = { ManagedBy = "terraform" } } delicense = { enabled = true ssm_param_name = "example_param_store_delicense" # TODO: update here } } }	map(object({ bootstrap_options = object({ mgmt-interface-swap = string plugin-op-commands = string panorama-server = string auth-key = string dgname = string tplname = string dhcp-send-hostname = string dhcp-send-client-id = string dhcp-accept-server-hostname = string dhcp-accept-server-domain = string }) panos_version = string ebs_kms_id = string vpc = string gwlb = string interfaces = map(object({ device_index = number security_group = string subnet = map(string) create_public_ip = bool source_dest_check = bool })) subinterfaces = map(map(object({ gwlb_endpoint = string subinterface = string }))) asg = object({ desired_cap = number min_size = number max_size = number lambda_execute_pip_install_once = bool }) scaling_plan = object({ enabled = bool metric_name = string target_value = number statistic = string cloudwatch_namespace = string tags = map(string) }) delicense = object({ enabled = bool ssm_param_name = string }) }))	{}	no
vpcs	A map defining VPCs with security groups and subnets. Following properties are available: - name: VPC name - cidr: CIDR for VPC - nacls: map of network ACLs - security_groups: map of security groups - subnets: map of subnets with properties: - az: availability zone - set: internal identifier referenced by main.tf - nacl: key of NACL (can be null) - routes: map of routes with properties: - vpc_subnet - built from key of VPCs concatenate with - and key of subnet in format: VPCKEY-SUBNETKEY - next_hop_key - must match keys use to create TGW attachment, IGW, GWLB endpoint or other resources - next_hop_type - internet_gateway, nat_gateway, transit_gateway_attachment or gwlbe_endpoint Example: vpcs = { example_vpc = { name = "example-spoke-vpc" cidr = "10.104.0.0/16" nacls = { trusted_path_monitoring = { name = "trusted-path-monitoring" rules = { allow_inbound = { rule_number = 300 egress = false protocol = "-1" rule_action = "allow" cidr_block = "0.0.0.0/0" from_port = null to_port = null } } } } security_groups = { example_vm = { name = "example_vm" rules = { all_outbound = { description = "Permit All traffic outbound" type = "egress", from_port = "0", to_port = "0", protocol = "-1" cidr_blocks = ["0.0.0.0/0"] } } } } subnets = { "10.104.0.0/24" = { az = "eu-central-1a", set = "vm", nacl = null } "10.104.128.0/24" = { az = "eu-central-1b", set = "vm", nacl = null } } routes = { vm_default = { vpc_subnet = "app1_vpc-app1_vm" to_cidr = "0.0.0.0/0" next_hop_key = "app1" next_hop_type = "transit_gateway_attachment" } } } }	map(object({ name = string cidr = string nacls = map(object({ name = string rules = map(object({ rule_number = number egress = bool protocol = string rule_action = string cidr_block = string from_port = string to_port = string })) })) security_groups = any subnets = map(object({ az = string set = string nacl = string })) routes = map(object({ vpc_subnet = string to_cidr = string next_hop_key = string next_hop_type = string })) }))	{}	no

Outputs

Name	Description
app_inspected_dns_name	FQDN of App Internal Load Balancer. Can be used in VM-Series configuration to balance traffic between the application instances.