Skip to main content

User-ID API Activity Monitor

note

In this tutorial we'll leverage GO's http library to implement a "Man in the Middle" that will behave as a reverse proxy between a User-ID enabled application and a PAN-OS NGFW.

User-ID API and External Dynamic List (EDL)​

They're probably the most common way to share external IP metadata with the NGFW. The Use cases for them are enormous: from blocking known DoS sources to forward traffic from specific IP addresses on low latency links.

Let's take for instance the following logs generated by the SSH daemon in a linux host.

Apr 12 16:00:40 raspberrypi sshd[8381]: Accepted publickey for xhoms from 192.168.1.31 port 56842 ssh2: RSA SHA256:umL2q3TS9quZ2y+3a7asRh+SuZSrwMeCuUPQTXTNQX0
Apr 12 16:00:40 raspberrypi sshd[8381]: pam_unix(sshd:session): session opened for user xhoms by (uid=0)
Apr 12 16:00:40 raspberrypi systemd-logind[282]: New session 2506 of user xhoms.
Apr 12 16:00:40 raspberrypi systemd: pam_unix(systemd-user:session): session opened for user xhoms by (uid=0)

Won't it be great to have a way to share the IP addresses used by the user xhoms with the NGFW so specific security policies are applied to traffic sourced by that address just because we now know the endpoint is being managed by that user?

The following Shell CGI script could be used to create an External Dynamic List (EDL) that would list all known IP addresses the account xhoms was used to login into this server.

cat /var/log/auth.log | sed -En 's/.*[[:space:]]sshd.*Accepted publickey for xhoms from ([^[:space:]]*).*/\1/p'

That is great but:

  • When would address be removed from the list? At the daily log rollout?
  • Are all listed IP addresses still being operated by the user xhoms?
  • Is there any way to remove addresses from the list at user logout?

To overcome these (and many other) limitations Palo Alto Networks NGFW feature a very powerful IP tagging API called User-ID. Although EDL and User-ID cover similar objectives there are fundamental technical differences between them:

  • User-ID is "asynchronous" (push mode) and provides a very flexible way to create groupings. Either by tagging address objects (Dynamic Address Group - DAG ) or by mapping users to addresses and then tagging these users (Dynamic User Group - DUG)
  • EDL is "universal". Almost all network appliance vendors provide a feature to fetch an IP address list from a URL.

Years ago it was up to the end customer to create its own connectors. Just like the CGI script we shared before. But as presence of PAN-OS powered NGFW's grown among enterprise customers more application vendors decided to leverage the User-ID value by providing API clients out-of-the-box. Althought this is great (for Palo Alto Network customers) losing the option to fetch a list from a URL (EDL mode) makes it more difficult to integrate legacy technologies that might be out there still in the network.

Hijacking User-ID messages with a proxy​

Let's assume you have an application that features a PAN-OS User-ID API client and that is capable of pushing log entries to the PAN-OS NGFW in an asynchronous way. Let's assume, as well, that there are still some legacy network devices in your network that need to fetch that address metadata from a URL. How difficult it would be to create a micro-service for that?

One option would be to leverage a PAN-OS XML SDK like PAN Python or PAN GO and to create a REST API that extracts the current state from the PAN-OS device using its OP API.

getting DAG state from PAN-OS XML API
GET https://<pan-os-device>/api
    ?key=<API-KEY>
    &type=op
    &cmd=<show><object><registered-ip><all></all></registered-ip></object></show>

HTTP/1.1 200 OK

<response status="success">
  <result>
    <entry ip="10.10.10.11" from_agent="0" persistent="1">
      <tag>
        <member>tag10</member>
      </tag>
    </entry>
    <entry ip="10.10.10.10" from_agent="0" persistent="1">
      <tag>
        <member>tag10</member>
      </tag>
    </entry>
    <count>2</count>
  </result>
</response>

It shouldn't be that difficult to parse the response and provide a plain list out of it. In fact, if you're in for a challenge, I'd let you think about using a XSLT processor to pipe it at the output of a cURL command and pack everything as a CGI script.

But in this tutorial we're going to levarage the GO Package https://pkg.go.dev/github.com/xhoms/panoslib/uid to follow a diferent approach: we'll hijack User-ID messages by implementing a micro-service that behaves as a reverse proxy between the application that features the User-ID client and the PAN-OS device. I like this approach because it opens the door to other use cases like enforcing timeout (adding timeout to entries that do not have it), converting DAG messages into DUG equivalents or adding additional tags based on the source application.

Building a reverse-proxy micro-service skeleton​

GO's http package is great. And its httputil package companion contains a ready-to-use reverse proxy type. Building the foundation of our tutorial is a 4-line code exercise.

hostport := "<panos-device>:443"
url, _ := url.Parse("https://" + hostport)
proxy := httputil.NewSingleHostReverseProxy(url)
http.ListenAndServe(":8080", proxy)

Next step is to provide a minimum of configuration options allowing the user to provide:

  • the target PAN-OS device (hostport) and
  • the TCP to bound our micro-service to (defaulting to 8080)

Let's then use the following version as the foundation for our reverse proxy application

func main() {
    var url *url.URL
    if hostport, exists := os.LookupEnv("TARGET"); exists {
        if target, err := url.Parse("https://target"); err == nil {
            log.Fatalf("https://%v is not a valid url", hostport)
        } else {
            url = target
        }
    } else {
        log.Fatal("mandatory TARGET env variable not provided")
    }
    port := "8080"
    if envport, exists := os.LookupEnv("PORT"); exists {
        port = envport
    }
    proxy := httputil.NewSingleHostReverseProxy(url)
    log.Printf("attempting to start micro-service on %v", port)
    log.Fatal(http.ListenAndServe(":"+port, proxy))
}

Monitoring the User-ID messages​

Next goal is to be able to monitor the User-ID messages sent to the PAN-OS device. To acomplish that we need to capture transactions that conform to the following schema:

  • pattern equals /api/
  • methods GET or PUT
  • parameter type (either in Query String or POST payload) equals to user-id

We'll leverage GO http's routing capability by creating two handlers. One of them will process all requests sent to /api/ while the other one will behave like a default route (/). In any case, the request will need to be forwarded to the PAN-OS device. So let's create a type that will hold the reverse proxy and that will implement the two handlers.

type manInMiddle struct {
    proxy *httputil.ReverseProxy
}

func newManInMiddle(url *url.URL) (m manInMiddle) {
    m = manInMiddle{
        proxy: httputil.NewSingleHostReverseProxy(url),
    }
    return
}

func (m manInMiddle) defaultHandler(w http.ResponseWriter, r *http.Request) {
    m.proxy.ServeHTTP(w, r)
}

func (m manInMiddle) apiHandler(w http.ResponseWriter, r *http.Request) {
    m.proxy.ServeHTTP(w, r)
}

And now let's change the application main's method final lines to configure routing instead of a dedicated handler for all requests.

//...
mim := newManInMiddle(url)
http.HandleFunc("/api/", mim.apiHandler)
http.HandleFunc("/", mim.defaultHandler)
log.Printf("attempting to start micro-service on %v", port)
log.Fatal(http.ListenAndServe(":"+port, nil))

Finally we need to implemente the apiHandler() to extract the User-ID messages (cmd parameter) by:

  • reading parameters from the URL Query String for GET requests or
  • performing the following tasks in case of POST requests
    1. read the request body and keep a safe copy
    2. re-set the body to be able to call ParseForm() and check if type equals user-id extracting cmd in such a case
    3. re-set the body to be able to invoke the reverse proxy

This is how the modified version of the apiHandler() looks like

func (m manInMiddle) apiHandler(w http.ResponseWriter, r *http.Request) {
    var err error
    if r.Method == http.MethodGet {
        if r.URL.Query().Get("type") == "user-id" {
            cmd := r.URL.Query().Get("cmd")
        }
    } else if r.Method == http.MethodPost {
        var body []byte
        if body, err = ioutil.ReadAll(r.Body); err == nil {
            r.Body = ioutil.NopCloser(bytes.NewReader(body))
            if err = r.ParseForm(); err == nil {
                r.Body = ioutil.NopCloser(bytes.NewReader(body))
                if r.Form.Get("type") == "user-id" {
                    cmd := r.Form.Get("cmd")
                }
            }
        }
    }
    if err == nil {
        m.proxy.ServeHTTP(w, r)
    } else {
        w.WriteHeader(http.StatusInternalServerError)
        w.Write([]byte(err.Error()))
    }
}

Creating the User-ID state​

Now it is time to:

  • parse the User-ID (XML) command we extracted and
  • keep a simulated stated from reading the different mappings (user-ip, user-group and ip-tag) and honouring the provided time out values for each of them.

Fortunatelly we can leverage the packages https://pkg.go.dev/github.com/xhoms/panoslib/collection, https://pkg.go.dev/github.com/xhoms/panoslib/uid and https://pkg.go.dev/github.com/xhoms/panoslib/uidmonitor to perform the heavy lifting.

The uid package features a User-ID payload builder that accepts many input formats, being an existing User-ID payload one of them. Final actions of the builder (i.e. UIDMessage() or Payload()) accept a type implementing the uid.Monitor interface. If present, then the Monitor's Log() method would be called for each item in the User-ID payload.

The uidmonitor package provides the MemMonitor type that implements the uid.Monitor interface by keeping a state of valid values in memory. MemMonitor features convenience methods to dump current state as IP lists.

Let's change our manInMiddle type and newManInMiddle() initialization to keep a MemMonitor variable.

type manInMiddle struct {
    mm    *uidmonitor.MemMonitor
    proxy *httputil.ReverseProxy
}

func newManInMiddle(url *url.URL) (m manInMiddle) {
    m = manInMiddle{
        mm:    uidmonitor.NewMemMonitor(),
        proxy: httputil.NewSingleHostReverseProxy(url),
    }
    return
}

And now let's implement a process() method that will drive our MemMonitor from hijacked UserID messages in the api handler.

func (m manInMiddle) process(cmd string) {
    var uidmsg *collection.UIDMessage = &collection.UIDMessage{}
    if err := xml.Unmarshal([]byte(cmd), uidmsg); err == nil {
        if uidmsg != nil {
            uid.NewBuilderFromPayload(uidmsg.Payload).
                Payload(m.mm)
        }
    }
}

Now we just need the apiHandler() to call process() for each User-ID command extracted by our Man-in-the-Middle.

//...
        if r.URL.Query().Get("type") == "user-id" {
            cmd := r.URL.Query().Get("cmd")
            m.process(cmd)
        }
//...
                if r.Form.Get("type") == "user-id" {
                    cmd := r.Form.Get("cmd")
                    m.process(cmd)
                }

The "virtual" /edl endpoint​

We're almost there. The only missing piece is a new handler capable of dumping our MemMonitor state. As that state is organized by type (TagIP(), UserIP() and GroupIP()) we will first implement a list() function to choose the right method with a switch/case statement.

func (m manInMiddle) list(edl string, key string) (out []string) {
    m.mm.CleanUp(time.Now())
    switch edl {
    case "user":
        out = m.mm.UserIP(key)
    case "group":
        out = m.mm.GroupIP(key)
    case "tag":
        out = m.mm.TagIP(key)
    default:
        out = emptyEdl
    }
    return
}

Notice we call MemMonitor's CleanUp() method with current time to have expired entries being removed before the list is retrieved.

And now let's wrap this list function in a http handler that will extract required parameters from a URL query string featuring the schema ?list=[user|group|tag]&key=<tag>

func (m manInMiddle) edlHandler(w http.ResponseWriter, r *http.Request) {
    out := &bytes.Buffer{}
    for _, item := range m.list(
        r.URL.Query().Get("list"),
        r.URL.Query().Get("key"),
    ) {
        out.WriteString(string(item) + "\n")
    }
    w.Header().Add("content-type", "text/plain")
    w.WriteHeader(http.StatusOK)
    w.Write(out.Bytes())
}

Final step is to insert this new handler in the http router. We can use the /edl endpoint as it won't conflict with current PAN-OS http service.

mim := newManInMiddle(url, false)
http.HandleFunc("/edl", mim.edlHandler)
http.HandleFunc("/api/", mim.apiHandler)
http.HandleFunc("/", mim.defaultHandler)
log.Printf("attempting to start micro-service on %v", port)
log.Fatal(http.ListenAndServe(":"+port, nil))

Our micro-service in action​

Let's perform a couple of test. First one will register two IP addresses with the tag test. One of these entries would expire in 100 seconds while the second one would expire in just 10 seconds.

<uid-message>
    <type>update</type>
    <payload>
        <register>
            <entry ip="10.10.10.10">
                <tag>
                    <member timeout="100">test</member>
                </tag>
            </entry>
            <entry ip="10.10.20.20">
                <tag>
                    <member timeout="10">test</member>
                </tag>
            </entry>
        </register>
    </payload>
</uid-message>

Then we call the /edl endpoint twice giving enought time between attempts to allow one of the entries to expire.

GET http://127.0.0.1:8080/edl/
    ?list=tag
    &key=test

HTTP/1.1 200 OK
Content-Type: text/plain
Date: Mon, 12 Apr 2021 10:07:50 GMT
Content-Length: 24
Connection: close

10.10.20.20
10.10.10.10

...

GET http://127.0.0.1:8080/edl/
    ?list=tag
    &key=test

Content-Type: text/plain
Date: Mon, 12 Apr 2021 10:08:55 GMT
Content-Length: 12
Connection: close

10.10.10.10

The next payload is a bit more complex. It maps (login) two users and then tags them to the group admin with different timeouts.

<uid-message>
    <type>update</type>
    <payload>
        <register-user>
            <entry user="foo@test.local">
                <tag>
                    <member timeout="100">admin</member>
                </tag>
            </entry>
            <entry user="bar@test.local">
                <tag>
                    <member timeout="10">admin</member>
                </tag>
            </entry>
        </register-user>
        <login>
            <entry name="bar@test.local" ip="10.10.20.20" timeout="100"></entry>
            <entry name="foo@test.local" ip="10.10.10.10" timeout="100"></entry>
        </login>
    </payload>
</uid-message>

Again, we can experience one of the IP addresses being removed from the output at its due time.

GET http://127.0.0.1:8080/edl/
    ?list=group
    &key=admin

HTTP/1.1 200 OK
Content-Type: text/plain
Date: Mon, 12 Apr 2021 10:28:16 GMT
Content-Length: 24
Connection: close

10.10.20.20
10.10.10.10

...

GET http://127.0.0.1:8080/edl/
    ?list=group
    &key=admin

HTTP/1.1 200 OK
Content-Type: text/plain
Date: Mon, 12 Apr 2021 10:28:32 GMT
Content-Length: 12
Connection: close

10.10.10.10

Code Repository and Container Image​

Feel free to clone the the GitHub repository created to host this tutorial code or run the linked Container Image for an out-of-the-box experience.

docker run --rm -p 8080:8080 -e TARGET=<pan-os-device> ghcr.io/xhoms/uidmonitor