Skip to main content

Reference Architecture with Terraform: VM-Series in AWS, Isolated Design Model, Common NGFW option with Autoscaling

Palo Alto Networks produces several validated reference architecture design and deployment documentation guides, which describe well-architected and tested deployments. When deploying VM-Series in a public cloud, the reference architectures guide users toward the best security outcomes, whilst reducing rollout time and avoiding common integration efforts. The Terraform code presented here will deploy Palo Alto Networks VM-Series firewalls in AWS based on the centralized design; for a discussion of other options, please see the design guide from the reference architecture guides.

GitHub Logo Terraform Logo

Reference Architecture Design​

Simplified High Level Topology Diagram

This code implements:

  • an isolated design, which secures outbound and inbound traffic flows using AWS Gateway Load Balancer (GWLB). Application resources are segmented across multiple VPCs that distribute traffic to the dedicated VPC for security services where the VM-Series are deployed.

Detailed Architecture and Design​

Isolated Design​

The Isolated Design model centralizes the security instances in a dedicated security VPC, while providing one or more isolated VPCs inbound and outbound security services. This design leverages a VPC dedicated to security. In the security VPC, you deploy the VM-Series firewalls, in separate availability zones, and a GWLB to distribute traffic to the firewalls. This design uses overlay routing for outbound security on the VM-Series firewalls. Outbound traffic from instances in the isolated VPCs uses the PrivateLink connections from GWLB endpoints in the applications. VPCs to the GWLB in the security VPC to egress the AWS environment through the VM-Series firewalls.

Inbound traffic originates outside the VPC and is destined to applications or services hosted within your VPCs, such as web servers. This design uses the GWLB and VM-Series firewalls in the security VPC, with GWLB endpoints in the application VPCs for the transparent inspection of inbound traffic.

image

Auto Scaling VM-Series​

Auto scaling: Public-cloud environments focus on scaling out a deployment instead of scaling up. This architectural difference stems primarily from the capability of public-cloud environments to dynamically increase or decrease the number of resources allocated to your environment. Using native AWS services like CloudWatch, auto scaling groups (ASG) and VM-Series automation features, the guide implements VM-Series that will scale in and out dynamically, as your protected workload demands fluctuate. The VM-Series firewalls are deployed in an auto scaling group, and are automatically registered to a Gateway Load Balancer. While bootstrapping the VM-Series, there are associations made automatically between VM-Series subinterfaces and the GWLB endpoints. Each VM-Series contains multiple network interfaces created by an AWS Lambda function.

Prerequisites​

The following steps should be followed before deploying the Terraform code presented here.

  1. Deploy Panorama e.g. by using Panorama example
  2. Prepare device group, template, template stack in Panorama
  3. Download and install plugin sw_fw_license for managing licenses
  4. Configure bootstrap definition and license manager
  5. Configure license API key
  6. Configure security rules and NAT rules for outbound traffic
  7. Configure interface management profile to enable health checks from GWLB
  8. Configure network interfaces and subinterfaces, zones and virtual router in template
  9. Configure static routes with path monitoring
  10. Configure VPC peering between VPC with Panorama and VPC with VM-Series in autoscaling group (after deploying that example)

Details - static routes with path monitoring​

Using multiple template stacks, one for each AZ complicates autoscaling and the Panorama Licensing plugin configuration. The virtual router (VR) configuration combined with path monitoring outlined below avoids using AZ-specific template stacks and variables.

Virtual Router Configuration

  1. Create static routes for all internally routed CIDRs
  2. Set the next hop to the default gateway IP of the trust subnet of the corresponding availability zone, which the firewall is connected to.
  3. Set a unique metric value per AZ so that it doesn't overlap with other routes with the same destinations.
  4. Enable Path Monitoring for the route.
  • Source IP: DHCP
  • Destination IP: Next Hop IP of the subnet of the corresponding AZ.

The AWS NACL applied to the trust subnets blocks the path monitor from pinging default gateways of the trust subnets in the other availability zones. This will cause the firewall to remove all routes that don't apply to the Availability zone it is in.

Below there is shown example of VR configuration with static routes and path monitoring:

NameDestinationNext HopMetricPath Monitor Destination IP
app1_az110.104.0.0/1610.100.1.11110.100.1.1
app2_az110.105.0.0/1610.100.1.11110.100.1.1
app1_az210.104.0.0/1610.100.65.11210.100.65.1
app2_az210.105.0.0/1610.100.65.11210.100.65.1
health_az110.100.0.0/1610.100.1.11110.100.1.1
health_az210.100.0.0/1610.100.65.11210.100.65.1

An example XML configuration snippet (for PANOS 10.2.3) of the described configuration can be found here, which after importing to Panorama, can be merged using the command:

load config partial mode merge from-xpath /config/devices/entry/template/entry[@name='asg'] to-xpath /config/devices/entry/template/entry[@name='asg'] from template-asg-path-monitoring.xml

Usage​

  1. Copy example.tfvars into terraform.tfvars
  2. Review terraform.tfvars file, especially with lines commented by # TODO: update here
  3. Initialize Terraform: terraform init
  4. Prepare plan: terraform plan
  5. Deploy infrastructure: terraform apply -auto-approve
  6. Destroy infrastructure if needed: terraform destroy -auto-approve

Additional Reading​

Lambda function​

Lambda function is used to handle correct lifecycle action:

  • instance launch or
  • instance terminate

In case of creating VM-Series, there are performed below actions, which cannot be achieved in AWS launch template:

  • change setting source_dest_check for first network interface (data plane)
  • setup additional network interfaces (with optional possibility to attach EIP)

In case of destroying VM-Series, there is performed below action:

  • clean EIP

Moreover having Lambda function executed while scaling out or in gives more options for extension e.g. delicesning VM-Series just after terminating instance.

Autoscaling​

AWS Auto Scaling monitors VM-Series and automatically adjusts capacity to maintain steady, predictable performance at the lowest possible cost. For autoscaling there are 10 metrics available from vmseries plugin:

  • DataPlaneCPUUtilizationPct
  • DataPlanePacketBufferUtilization
  • panGPGatewayUtilizationPct
  • panGPGWUtilizationActiveTunnels
  • panSessionActive
  • panSessionConnectionsPerSecond
  • panSessionSslProxyUtilization
  • panSessionThroughputKbps
  • panSessionThroughputPps
  • panSessionUtilization

Using that metrics there can be configured different scaling plans. Below there are some examples, which can be used. All examples are based on target tracking configuration in scaling plan. Below code is already embedded into asg module:

  scaling_instruction {
    max_capacity       = var.max_size
    min_capacity       = var.min_size
    resource_id        = format("autoScalingGroup/%s", aws_autoscaling_group.this.name)
    scalable_dimension = "autoscaling:autoScalingGroup:DesiredCapacity"
    service_namespace  = "autoscaling"
    target_tracking_configuration {
      customized_scaling_metric_specification {
        metric_name = var.scaling_metric_name
        namespace   = var.scaling_cloudwatch_namespace
        statistic   = var.scaling_statistic
      }
      target_value = var.scaling_target_value
    }
  }

Using metrics from vmseries plugin we can defined multiple scaling configurations e.g.:

  • based on number of active sessions:
metric_name  = "panSessionActive"
target_value = 75
statistic    = "Average"
  • based on data plane CPU utilization and average value above 75%:
metric_name  = "DataPlaneCPUUtilizationPct"
target_value = 75
statistic    = "Average"
  • based on data plane packet buffer utilization and max value above 80%
metric_name  = "DataPlanePacketBufferUtilization"
target_value = 80
statistic    = "Maximum"

Reference​

Requirements​

NameVersion
terraform>= 1.0.0, < 2.0.0
aws~> 5.17

Providers​

NameVersion
aws~> 5.17

Modules​

NameSourceVersion
gwlb../../modules/gwlbn/a
gwlbe_endpoint../../modules/gwlb_endpoint_setn/a
public_alb../../modules/albn/a
public_nlb../../modules/nlbn/a
subnet_sets../../modules/subnet_setn/a
vm_series_asg../../modules/asgn/a
vpc../../modules/vpcn/a
vpc_routes../../modules/vpc_routen/a

Resources​

NameType
aws_iam_instance_profile.spoke_vm_iam_instance_profileresource
aws_iam_instance_profile.vm_series_iam_instance_profileresource
aws_iam_role.spoke_vm_ec2_iam_roleresource
aws_iam_role.vm_series_ec2_iam_roleresource
aws_iam_role_policy.vm_series_ec2_iam_policyresource
aws_instance.spoke_vmsresource
aws_vpc_peering_connection.thisresource
aws_ami.thisdata source
aws_caller_identity.thisdata source
aws_ebs_default_kms_key.currentdata source
aws_kms_alias.current_arndata source
aws_partition.thisdata source

Inputs​

NameDescriptionTypeDefaultRequired
global_tagsGlobal tags configured for all provisioned resourcesanyn/ayes
gwlb_endpointsA map defining GWLB endpoints.

Following properties are available:
- name: name of the GWLB endpoint
- gwlb: key of GWLB
- vpc: key of VPC
- vpc_subnet: key of the VPC and subnet connected by '-' character
- act_as_next_hop: set to true if endpoint is part of an IGW route table e.g. for inbound traffic
- to_vpc_subnets: subnets to which traffic from IGW is routed to the GWLB endpoint

Example:
gwlb_endpoints = {
  security_gwlb_eastwest = {
    name            = "eastwest-gwlb-endpoint"
    gwlb            = "security_gwlb"
    vpc             = "security_vpc"
    vpc_subnet      = "security_vpc-gwlbe_eastwest"
    act_as_next_hop = false
    to_vpc_subnets  = null
  }
}
map(object({
    name            = string
    gwlb            = string
    vpc             = string
    vpc_subnet      = string
    act_as_next_hop = bool
    to_vpc_subnets  = string
  }))
{}no
gwlbsA map defining Gateway Load Balancers.

Following properties are available:
- name: name of the GWLB
- vpc_subnet: key of the VPC and subnet connected by '-' character

Example:
gwlbs = {
  security_gwlb = {
    name       = "security-gwlb"
    vpc_subnet = "security_vpc-gwlb"
  }
}
map(object({
    name       = string
    vpc_subnet = string
  }))
{}no
name_prefixPrefix used in names for the resources (VPCs, EC2 instances, autoscaling groups etc.)stringn/ayes
panorama_connectionA object defining VPC peering and CIDR for Panorama.

Following properties are available:
- security_vpc: key of the security VPC
- peering_vpc_id: ID of the VPC for Panorama
- vpc_cidr: CIDR of the VPC, where Panorama is deployed

Example:
panorama = {
  security_vpc   = "security_vpc"
  peering_vpc_id = "vpc-1234567890"
  vpc_cidr       = "10.255.0.0/24"
}
object({
    security_vpc   = string
    peering_vpc_id = string
    vpc_cidr       = string
  })
nullno
regionAWS region used to deploy whole infrastructurestringn/ayes
spoke_albsA map defining Application Load Balancers deployed in spoke VPCs.

Following properties are available:
- rules: Rules defining the method of traffic balancing
- vms: Instances to be the target group for ALB
- vpc: The VPC in which the load balancer is to be run
- vpc_subnet: The subnets in which the Load Balancer is to be run
- security_gropus: Security Groups to be associated with the ALB
map(object({
    rules           = any
    vms             = list(string)
    vpc             = string
    vpc_subnet      = string
    security_groups = string
  }))
n/ayes
spoke_nlbsA map defining Network Load Balancers deployed in spoke VPCs.

Following properties are available:
- vpc_subnet: key of the VPC and subnet connected by '-' character
- vms: keys of spoke VMs

Example:
spoke_lbs = {
  "app1-nlb" = {
    vpc_subnet = "app1_vpc-app1_lb"
    vms        = ["app1_vm01", "app1_vm02"]
  }
}
map(object({
    vpc_subnet = string
    vms        = list(string)
  }))
{}no
spoke_vmsA map defining VMs in spoke VPCs.

Following properties are available:
- az: name of the Availability Zone
- vpc: name of the VPC (needs to be one of the keys in map vpcs)
- vpc_subnet: key of the VPC and subnet connected by '-' character
- security_group: security group assigned to ENI used by VM
- type: EC2 type VM

Example:
spoke_vms = {
  "app1_vm01" = {
    az             = "eu-central-1a"
    vpc            = "app1_vpc"
    vpc_subnet     = "app1_vpc-app1_vm"
    security_group = "app1_vm"
    type           = "t2.micro"
  }
}
map(object({
    az             = string
    vpc            = string
    vpc_subnet     = string
    security_group = string
    type           = string
  }))
{}no
ssh_key_nameName of the SSH key pair existing in AWS key pairs and used to authenticate to VM-Series or test boxesstringn/ayes
vmseries_asgsA map defining Autoscaling Groups with VM-Series instances.

Following properties are available:
- bootstrap_options: VM-Seriess bootstrap options used to connect to Panorama
- panos_version: PAN-OS version used for VM-Series
- ebs_kms_id: alias for AWS KMS used for EBS encryption in VM-Series
- vpc: key of VPC
- gwlb: key of GWLB
- interfaces: configuration of network interfaces for VM-Series used by Lamdba while provisioning new VM-Series in autoscaling group
- subinterfaces: configuration of network subinterfaces used to map with GWLB endpoints
- asg: the number of Amazon EC2 instances that should be running in the group (desired, minimum, maximum)
- scaling_plan: scaling plan with attributes
- enabled: true if automatic dynamic scaling policy should be created
- metric_name: name of the metric used in dynamic scaling policy
- target_value: target value for the metric used in dynamic scaling policy
- statistic: statistic of the metric. Valid values: Average, Maximum, Minimum, SampleCount, Sum
- cloudwatch_namespace: name of CloudWatch namespace, where metrics are available (it should be the same as namespace configured in VM-Series plugin in PAN-OS)
- tags: tags configured for dynamic scaling policy

Example:
vmseries_asgs = {
  main_asg = {
    bootstrap_options = {
      mgmt-interface-swap         = "enable"
      plugin-op-commands          = "panorama-licensing-mode-on,aws-gwlb-inspect:enable,aws-gwlb-overlay-routing:enable" # TODO: update here
      panorama-server             = ""                                                                                   # TODO: update here
      auth-key                    = ""                                                                                   # TODO: update here
      dgname                      = ""                                                                                   # TODO: update here
      tplname                     = ""                                                                                   # TODO: update here
      dhcp-send-hostname          = "yes"                                                                                # TODO: update here
      dhcp-send-client-id         = "yes"                                                                                # TODO: update here
      dhcp-accept-server-hostname = "yes"                                                                                # TODO: update here
      dhcp-accept-server-domain   = "yes"                                                                                # TODO: update here
    }

    panos_version = "10.2.3"        # TODO: update here
    ebs_kms_id    = "alias/aws/ebs" # TODO: update here

    vpc               = "security_vpc"
    gwlb              = "security_gwlb"

    interfaces = {
      private = {
        device_index   = 0
        security_group = "vmseries_private"
        subnet = {
          "privatea" = "eu-central-1a",
          "privateb" = "eu-central-1b"
        }
        create_public_ip  = false
        source_dest_check = false
      }
      mgmt = {
        device_index   = 1
        security_group = "vmseries_mgmt"
        subnet = {
          "mgmta" = "eu-central-1a",
          "mgmtb" = "eu-central-1b"
        }
        create_public_ip  = true
        source_dest_check = true
      }
      public = {
        device_index   = 2
        security_group = "vmseries_public"
        subnet = {
          "publica" = "eu-central-1a",
          "publicb" = "eu-central-1b"
        }
        create_public_ip  = false
        source_dest_check = false
      }
    }

    subinterfaces = {
      inbound = {
        app1 = {
          gwlb_endpoint = "app1_inbound"
          subinterface  = "ethernet1/1.11"
        }
        app2 = {
          gwlb_endpoint = "app2_inbound"
          subinterface  = "ethernet1/1.12"
        }
      }
      outbound = {
        only_1_outbound = {
          gwlb_endpoint = "security_gwlb_outbound"
          subinterface  = "ethernet1/1.20"
        }
      }
      eastwest = {
        only_1_eastwest = {
          gwlb_endpoint = "security_gwlb_eastwest"
          subinterface  = "ethernet1/1.30"
        }
      }
    }

    asg = {
      desired_cap                     = 0
      min_size                        = 0
      max_size                        = 4
      lambda_execute_pip_install_once = true
    }

    scaling_plan = {
      enabled              = true               # TODO: update here
      metric_name          = "panSessionActive" # TODO: update here
      target_value         = 75                 # TODO: update here
      statistic            = "Average"          # TODO: update here
      cloudwatch_namespace = "example-vmseries" # TODO: update here
      tags = {
        ManagedBy = "terraform"
      }
    }
  }
}
map(object({
    bootstrap_options = object({
      mgmt-interface-swap         = string
      plugin-op-commands          = string
      panorama-server             = string
      auth-key                    = string
      dgname                      = string
      tplname                     = string
      dhcp-send-hostname          = string
      dhcp-send-client-id         = string
      dhcp-accept-server-hostname = string
      dhcp-accept-server-domain   = string
    })

    panos_version = string
    ebs_kms_id    = string

    vpc  = string
    gwlb = string

    interfaces = map(object({
      device_index      = number
      security_group    = string
      subnet            = map(string)
      create_public_ip  = bool
      source_dest_check = bool
    }))

    subinterfaces = map(map(object({
      gwlb_endpoint = string
      subinterface  = string
    })))

    asg = object({
      desired_cap                     = number
      min_size                        = number
      max_size                        = number
      lambda_execute_pip_install_once = bool
    })

    scaling_plan = object({
      enabled              = bool
      metric_name          = string
      target_value         = number
      statistic            = string
      cloudwatch_namespace = string
      tags                 = map(string)
    })
  }))
{}no
vpcsA map defining VPCs with security groups and subnets.

Following properties are available:
- name: VPC name
- cidr: CIDR for VPC
- nacls: map of network ACLs
- security_groups: map of security groups
- subnets: map of subnets with properties:
- az: availability zone
- set: internal identifier referenced by main.tf
- nacl: key of NACL (can be null)
- routes: map of routes with properties:
- vpc_subnet - built from key of VPCs concatenate with - and key of subnet in format: VPCKEY-SUBNETKEY
- next_hop_key - must match keys use to create TGW attachment, IGW, GWLB endpoint or other resources
- next_hop_type - internet_gateway, nat_gateway, transit_gateway_attachment or gwlbe_endpoint

Example:
vpcs = {
  example_vpc = {
    name = "example-spoke-vpc"
    cidr = "10.104.0.0/16"
    nacls = {
      trusted_path_monitoring = {
        name               = "trusted-path-monitoring"
        rules = {
          allow_inbound = {
            rule_number = 300
            egress      = false
            protocol    = "-1"
            rule_action = "allow"
            cidr_block  = "0.0.0.0/0"
            from_port   = null
            to_port     = null
          }
        }
      }
    }
    security_groups = {
      example_vm = {
        name = "example_vm"
        rules = {
          all_outbound = {
            description = "Permit All traffic outbound"
            type        = "egress", from_port = "0", to_port = "0", protocol = "-1"
            cidr_blocks = ["0.0.0.0/0"]
          }
        }
      }
    }
    subnets = {
      "10.104.0.0/24"   = { az = "eu-central-1a", set = "vm", nacl = null }
      "10.104.128.0/24" = { az = "eu-central-1b", set = "vm", nacl = null }
    }
    routes = {
      vm_default = {
        vpc_subnet    = "app1_vpc-app1_vm"
        to_cidr       = "0.0.0.0/0"
        next_hop_key  = "app1"
        next_hop_type = "transit_gateway_attachment"
      }
    }
  }
}
map(object({
    name = string
    cidr = string
    nacls = map(object({
      name = string
      rules = map(object({
        rule_number = number
        egress      = bool
        protocol    = string
        rule_action = string
        cidr_block  = string
        from_port   = string
        to_port     = string
      }))
    }))
    security_groups = any
    subnets = map(object({
      az   = string
      set  = string
      nacl = string
    }))
    routes = map(object({
      vpc_subnet    = string
      to_cidr       = string
      next_hop_key  = string
      next_hop_type = string
    }))
  }))
{}no

Outputs​

NameDescription
application_load_balancersFQDNs of Application Load Balancers
network_load_balancersFQDNs of Network Load Balancers.