Export security rule hitcount to CSV
This tutorial covers how to dump a security rulebase hit count into a CSV file for offline processing. A typical use case leveraging rule hit counts is to identify rules with zero or very few hits that, otherwise, might be eligible for cleanup or deletion.
Requirements
To follow this tutorial, it is recommended that that you are familiar with the concepts of Palo Alto Networks Next-Generation Firewalls, Security Policies and APIs. Some basic understanding of XML is also recommended.
Make sure you have a Palo Alto Networks Next-Generation Firewall deployed and that you have administrative access to its Management interface via HTTPS. To avoid potential disruptions, it's recommended to run all the tests on a non-production environment.
In this tutorial you'll find code examples leveraging the pan-python and pan-go. It is assumed that the reader has basic PAN-OS API knowledge (curl) and that owns an API KEY for his PAN-OS device (Grab the API Key)
Rule hit count
Starting with PAN-OS 8.1, the firewall web and command line interface displays the hit count and additional metadata for traffic matching rules in different rulesets.
These runtime statistics can provide value in some automation use cases. For instance:
- In large deployments (thousands of rules) it might be desirable to export specific datapoints (i.e. rule name, hit count and last hit timestamp) into a CSV file for out-of-band processing (periodic security audit checks)
- SOC operations might be interested on displaying and analyzing time-based graphs of hit-count on selected rules.
Extracting runtime statistics using PAN-OS API
The above use cases justify the need for a programmatic way of extracting runtime data points (the rule hit count in this case) from the PAN-OS device.
A network security operations engineer might be accustomed to using the device's CLI (PAN-OS CLI in this case) to access that data. The following is an example of CLI command displaying the rule hit count on a Palo Alto Networks firewall.
xhoms@PA-220> show rule-hit-count vsys vsys-name vsys1 rule-base security rules all
Rule Name Hit Count Last Hit Timestamp Last Reset Timestamp First Hit Timestamp Rule Create Timestamp Rule Modify Timestamp
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
LogAllURL-DIdac 510932 Mon Dec 17 15:42:45 2018 - Mon Jun 18 15:22:31 2018 Mon Jun 18 15:22:31 2018 Mon Jun 18 15:22:31 2018
HostedPanorama 4487 Fri Apr 27 11:21:18 2018 - Wed Apr 25 11:40:01 2018 Wed Apr 25 11:40:01 2018 Wed Apr 25 11:40:01 2018
Minemeld hosted on Rancher 180262 Wed Nov 20 12:58:01 2019 - Fri Sep 13 12:06:27 2019 Fri Sep 13 11:35:24 2019 Fri Sep 13 12:04:04 2019
ApolloLCaaS 40945810 Wed Nov 20 13:51:02 2019 - Tue Apr 10 18:49:52 2018 Tue Apr 10 18:49:52 2018 Tue Apr 10 18:49:52 2018
CrashPlan Access 29184 Wed Nov 20 11:31:07 2019 - Thu Jan 24 15:52:04 2019 Thu Jan 24 15:52:04 2019 Thu Jan 24 15:52:04 2019
VPN Xavi Back 2321 Wed Jan 17 07:27:20 2018 - Fri Jan 12 22:40:00 2018 Fri Jan 12 22:40:00 2018 Fri Jan 12 22:40:00 2018
...
For years, the only way to programmatically access this type of data was using CLI automation tools (Expect / Puppet / ...). But, nowadays, most teams look to leverage APIs to support automation use cases.
The "type=op" PAN-OS API command
PAN-OS API supports many different type of requests:
type=op(operation commands: i.e. show commands)type=config(configuration management)type=log(get log events)type=user-id(dataplane real-time object update)type=keygen(generate an API KEY out of user and password data)type=report(request report generation)- ... and few other types
The case we're covering in this tutorial requires us to use a type=op API requests. Examples of CLI commands that can be emulated with API type=op requests are show, clear, delete, test, request, etc.
type=op API requests requires a mandatory cmd=<xml-command> parameter. <xml-command> is a XML document describing the command to be executed.
For fluent PAN-OS CLI engineers, the easiest way to get the XML document of a CLI command is by enablig CLI debug on their terminal session as in the following example:
debug cli on
show rule-hit-count vsys vsys-name vsys1 rule-base security rules all
Example output:
(container-tag: rule-hit-count container-tag: vsys container-tag: vsys-name container-tag: entry key-tag: name value: vsys1 container-tag: rule-base container-tag: entry key-tag: name value: security container-tag: rules container-tag: all pop-tag: pop-tag: pop-tag: pop-tag: pop-tag: pop-tag: pop-tag: pop-tag:)
((eol-matched: . #t) (context-inserted-at-end-p: . #f))
<request cmd="op" cookie="2849954914780104" uid="1001"><operations><show><rule-hit-count><vsys><vsys-name><entry name='vsys1'><rule-base><entry name='security'><rules><all/></rules></entry></rule-base></entry></vsys-name></vsys></rule-hit-count></show></operations></request>
...
The output of any CLI command after debug has been turned on will include two XML blocks. The first one contains data about the API request and the second one the raw XML response (in the example above only the first XML block is shown).
The request will look like <request><operations>[xml-command]</operations></request>
With all information we have so far we're ready to construct the URL for our API call. It would be:
https://<panos-device>/api/?key=<api-key>&type=op&cmd=<show><rule-hit-count><vsys><vsys-name><entry name='vsys1'><rule-base><entry name='security'><rules><all></all></rules></entry></rule-base></entry></vsys-name></vsys></rule-hit-count></show>
The following linux command sequence can be used to perform the previous API call:
HOSTNAME=10.11.29.250
API_KEY=LUFRPT1HR...
CMD=<show><rule-hit-count><vsys><vsys-name><entry name='vsys1'><rule-base><entry name='security'><rules><all></all></rules></entry></rule-base></entry></vsys-name></vsys></rule-hit-count></show>
curl -k --data-urlencode "cmd=$CMD" "https://$HOSTNAME/api/?key=$API_KEY&type=op"
Example response
<response status="success"><result><rule-hit-count><vsys><entry name="vsys1"><rule-base><entry name="security"><rules><entry name="LogAllURL-DIdac"><latest>yes</latest><hit-count>510932</hit-count><last-hit-timestamp>1545057765</last-hit-timestamp><last-reset-timestamp>0</last-reset-timestamp><first-hit-timestamp>1529328151</first-hit-timestamp><rule-creation-timestamp>1529328151</rule-creation-timestamp><rule-modification-timestamp>1529328151</rule-modification-timestamp></entry><entry name="HostedPanorama">...
The output is a large XML document (it contains hit count details for all rules in the security rule-set) with the following structure:
response
result
rule-hit-count
vsys
entry
rule-base
entry
rules
entry name=<rule-name>
hit-count
last-hit-timestamp
...
entry
hit-count
last-hit-timestamp
...
As you can see the job of extracting runtime data from a PAN-OS firewall using the API is quite straightforward. Processing the XML output, though, can be a bit tough using plain POSIX/GNU CLI tools.
Let's leverage two of the most used PAN-OS SDK's ( pan-python and pan-go ) to get the job done.
Exporting rule hit count to CSV using Python
Look at the following python3 snippet
import pan.xapi
import sys
import operator
import datetime
API_KEY = "LUFRPT1HR.."
HOSTNAME = "10.11.29.250"
CMD = "<show><rule-hit-count><vsys><vsys-name><entry name='vsys1'><rule-base><entry name='security'><rules><all></all></rules></entry></rule-base></entry></vsys-name></vsys></rule-hit-count></show>"
def main():
try:
xapi = pan.xapi.PanXapi(
api_key=API_KEY,
hostname=HOSTNAME,
use_get=True
)
except pan.xapi.PanXapiError as msg:
print('pan.xapi.PanXapi:', msg, file=sys.stderr)
sys.exit(1)
try:
# STEP 1: Execute the operational command
xapi.op(CMD)
# STEP 2: Parse the XML response
rules = xapi.element_root.iterfind('.//rules/entry')
rule_dict = {}
for rule in rules:
rule_name = rule.get('name')
hit_count = rule.findtext('hit-count')
last_hit_ts = rule.findtext('last-hit-timestamp')
rule_dict[rule_name] = (
int(hit_count), datetime.datetime.fromtimestamp(int(last_hit_ts)).isoformat())
# STEP 3: Sort the data set and print to stdout with CSV format
print("rulename,hitcount,lasthitts")
for rule_name, (hit_count, last_hit_ts) in sorted(rule_dict.items(), key=operator.itemgetter(1)):
print("%s,%s,%s" % (rule_name, hit_count, last_hit_ts))
except pan.xapi.PanXapiError as msg:
print('set:', msg, file=sys.stderr)
sys.exit(1)
if __name__ == "__main__":
main()
The first try / except block attemps to leverage pan-python to initialize a xapi object connected to our PAN-OS device API
The second try / except block performs the following steps in sequence:
- Execute the operational command
- Parse the XML response
- Process the result data set (order by hit count number) and print CSV to sdtout
Let me highline some pieces of code:
rules = xapi.element_root.iterfind('.//rules/entry')
pan-python's xapi object features automatic XML parsing of the API response. A field named element_root holds the xml.etree object corresponding to the parsed XML response.
That allows us to use the iterfind method to produce a generator (rules) that will yield all elements with XML tag entry at the XPATH .//rules/entry (look above for the XML response schema of this operational command)
rule_dict = {}
for rule in rules:
rule_name = rule.get('name')
hit_count = rule.findtext('hit-count')
last_hit_ts = rule.findtext('last-hit-timestamp')
rule_dict[rule_name] = (
int(hit_count), datetime.datetime.fromtimestamp(int(last_hit_ts)).isoformat())
A loop iterating the rules generator will convert the raw data into a python dictionary whose key will be the rule name and having the tuple (hit_count, last_hit_ts) as its value.
Inside the loop, rule becomes an xml.etree object that will be used to:
- extract the name XML attribute using its
get()method (rule name) - extract the XML text node from the XML elements hit-count and last-hit-timestamp using its
findtext()method
The XML text values are casted to integer (hit-count) and formatted (date representation of the UNIX timestamp)
The last step is a basic sort by hit-count and formatted print (CSV) to stdout.
Exporting rule hit count to CSV using GO
Look at the following GO snippet
package main
import (
"fmt"
"log"
"sort"
"time"
"github.com/PaloAltoNetworks/pango"
)
const (
apiKey = "LUFRPT1HR.."
hostname = "10.11.29.250"
cmd = "<show><rule-hit-count><vsys><vsys-name><entry name='vsys1'><rule-base><entry name='security'><rules><all></all></rules></entry></rule-base></entry></vsys-name></vsys></rule-hit-count></show>"
)
type rule struct {
RuleName string `xml:"name,attr"`
HitCount int64 `xml:"hit-count"`
LastHitTs int64 `xml:"last-hit-timestamp"`
}
type response struct {
Rules []rule `xml:"result>rule-hit-count>vsys>entry>rule-base>entry>rules>entry"`
}
type byHitCount []rule
func (r byHitCount) Len() int { return len(r) }
func (r byHitCount) Swap(i, j int) { r[i], r[j] = r[j], r[i] }
func (r byHitCount) Less(i, j int) bool { return r[i].HitCount < r[j].HitCount }
func main() {
var err error
var response = response{}
c := &pango.Firewall{Client: pango.Client{
Hostname: hostname,
ApiKey: apiKey,
}}
if err = c.Initialize(); err != nil {
log.Printf("Failed to initialize client: %s", err)
return
}
// STEP 1: Execute the operational command and get the unmarshalled XML response
_, respErr := c.Op(cmd, "", nil, &response)
if respErr != nil {
log.Printf("Failed to execute Op command: %s", respErr)
}
// STEP 2: Sort the data set and print to stdout with CSV format
sort.Sort(byHitCount(response.Rules))
fmt.Println("rulename,hitcount,lasthitts")
for _, entry := range response.Rules {
fmt.Printf("%v,%v,%v\n", entry.RuleName, entry.HitCount,
time.Unix(entry.LastHitTs, 0).Format(time.RFC3339))
}
}
Let's comment some parts of the code.
c := &pango.Firewall{Client: pango.Client{
Hostname: hostname,
ApiKey: apiKey,
}}
if err = c.Initialize(); err != nil {
log.Printf("Failed to initialize client: %s", err)
return
}
This is how you define a Firewall struct for your PAN-OS device and how you Initialize() it to be able to use its Op() function.
_, respErr := c.Op(cmd, "", nil, &response)
The Op(req interface{}, vsys string, extras interface{}, ans interface{}) method executes the API req (a string representing a XML command in this case) and returns a []byte slice with the XML response (we're discarding it). In case you provide a non-nil value to the ans parameter (we are providing &response) then the Op() function will try to unmarshall the XML response in your structure.
The following is the struct of the response we're passing to Op(). Using this strategy we leverage pan-go's XML parser and data conversion features simplifying data processing on our end.
type rule struct {
RuleName string `xml:"name,attr"`
HitCount int64 `xml:"hit-count"`
LastHitTs int64 `xml:"last-hit-timestamp"`
}
type response struct {
Rules []rule `xml:"result>rule-hit-count>vsys>entry>rule-base>entry>rules>entry"`
}
The last part of the code is a simple sort and CSV print to stdout code.
Summary
Many automation use cases require the DevOps engineers to be able to extract runtime data from PAN-OS firewalls. Any data that can obtained from CLI commands can also be accessed programmatically using the XML API.
Although the API can be consumed directly (i.e. using curl commands) it is highly recommended to leverage any available SDK. In this tutorial we've introduced both the pan-python and pan-go SDK's and highlined the benefits of using them (i.e. XML response parsing and data type conversion)