PCAP analysis IV: Putting it all together ·

Table of Contents

In previous posts we went thourgh the basic analytic process when reviewing pcap files. While it is usually quite easy to find what you are looking for using these methods, it is also true that most of the time we don’t know what we are looking for. So how do we figure out the essentials and get those anomalies which we can then dig deeper?

Automation
#

Let’s say that you have saved 10G of network traffic and you need to know what is inside those packets. It is not feasible to manually review them nor is it very easy to keep track of what is going on. So easiest solution is to automate the whole process so you can focus on analysis and computer can crunch the data.

What should automation do?

Open pcap
Read it into memory so we don’t have to utilize harddisk
Count available cpus
Start analysis process on each available cpu and log results
Wait until analysis is complete and start another process
Wait until all analytics have finnished
Repeat

So, quite simple. Now all we have to do is write it down and test it out :-)

You can find latest version of this script https://github.com/jerQ/pat

#!/bin/bash
# PCAP-ANALYSIS
# Created: jerq <[email protected]>
# 
# Usage:
# Run this script in same directory where pcap-files are located

# Functions
# Protocol Hierarchy Statistics
phs () {
  tshark -r $1 -qz io,phs 2>/dev/null >> $2.protocol-hierarchy.log
}

# DNS Tree
dns_tree () {
  tshark -r $1 -qz dns,tree 2>/dev/null >> $2.dns_tree.log
}

# TCP Streams
tcp_streams () {
  tshark -r $1 -qz conv,tcp 2>/dev/null >> $2.tcp_streams.log
}

# UDP Streams
udp_streams () {
  tshark -r $1 -qz conv,udp 2>/dev/null >> $2.udp_streams.log
}

# Unique IP-addresses
uniq_ips () {
  tshark -r $1 -T fields -e ip.dst ip.src 2>/dev/null >> $2.uniq_ip.log
}

# DNS query names
dns_queries () {
  tshark -r $1 -e ip.src -e dns.qry.name -e dns.a -T fields dns 2>/dev/null >> $2.dns_queries.log
}

# IP conversation partners
conv_ip () {
  tshark -r $1 -qz conv,ip 2>/dev/null >> $2.conv_ip.log
}

# Conversation partners TCP
conv_partners_tcp () {
  tshark -r $1 -T fields -e ip.src -e ip.dst -e tcp.dstport 2>/dev/null >> $2.tcp_conv_partners.log
}

# Conversation partners UDP
conv_partners_udp () {
  tshark -r $1 -T fields -e ip.src -e ip.dst -e udp.dstport 2>/dev/null >> $2.udp_conv_partners.log
}

# HTTP hosts
http_hosts () {
  tshark -r $1 -T fields -e http.host 2>/dev/null >> $2.http_hosts.log
}

# User Agents
user_agents () {
  tshark -r $1 -T fields -e http.user_agent 2>/dev/null >> $2.user_agent.log
}

# Endpoints
endpoints_ip () {
  tshark -r $1 -qz endpoints,ip 2>/dev/null >> $2.endpoints_ip.log
}

# Check if processors are fully utilized
check_jobs () {
  if [[ $(jobs -r -p | wc -l) -ge $cpus ]]; then
    wait -n
  fi
}


# Main program

# Find PCAP-files from local directory
for filecap in $(ls -1 *.pcap); do
  # Create tempfile to memory
  memcap="$(mktemp -p /dev/shm)"

  # Read pcap to mem
  tshark -r $filecap -w $memcap 2>/dev/null

  # Number of CPUs
  cpus=$(grep -c ^processor /proc/cpuinfo)

  # Output file
  logfile=$(echo "${filecap##*/}")

  # Analyze each file
  phs $memcap $logfile &
  check_jobs 

  dns_tree $memcap $logfile &
  check_jobs 

  tcp_streams $memcap $logfile &
  check_jobs 

  udp_streams $memcap $logfile &
  check_jobs 

  uniq_ips $memcap $logfile &
  check_jobs 

  dns_queries $memcap $logfile &
  check_jobs 

  conv_ip $memcap $logfile &
  check_jobs 

  conv_partners_tcp $memcap $logfile &
  check_jobs 

  conv_partners_udp $memcap $logfile &
  check_jobs 

  http_hosts $memcap $logfile &
  check_jobs 

  user_agents $memcap $logfile &
  check_jobs 

  endpoints_ip $memcap $logfile &
  check_jobs 

  # Wait until all processes are finnished
  # Then move to next pcap
  wait -n

  # Clean up and remove memcap
  rm $memcap
done

Ok, that’s it. Hopefully you got something out of this and I’ll see you in next post.

Automation #

Automation
#