Almost a year ago we introduced a new feature: Call recording using PCAP.For those who are not familiar with the term PCAP, here is a short explanation (more to be found on http://en.wikipedia.org/wiki/Pcap).

For us the PBX guys, this is a file format that we can generate that contains the packets that are processed on the PBX. There are several tools available out there that can read PCAP and display it in various formats, including Wireshark. The great things about using such tools is that they have special functions specifically for SIP and RTP. This way, whoever needs to look at problematic calls, can drill down to each bit of the call. Together with the timestamps, this provides an incredible level of detail that can still be managed easily.

There were two milestone achievement that we did with the PCAP introduction. The first one is that each call gets its own PCAP file. That is usually not possible when using external tools. They usually record everything on a network interface. When doing this, you get a lot of by-catch that is irrelevant for the analysis of the call. And you have to sort out which packet belongs to which call. Whoever has done this on a busy server knows what I am talking about. File sizes can easily get to a gigabyte or more, and then every time filtering for a call stalls the PC for a minute or two.

The other huge achievement that is that the PBX has the security context for generating decrypted PCAP files, even if the call was encrypted. This makes it possible to use the PCAP recording in live environments where TLS and SRTP are essentially and cannot be turned off for the sake of troubleshooting.

Because of the decrypting of the SIP packets, TLS traffic is stored as UDP packets. It looks kind of weird at first glance when you see a UDP packet that has the SIP/2.0/TLS in it. It was the easiest way to get the packet into the PCAP. The problem is that TLS packets can get big, like TCP packets. Unfortunately, it crashes Wireshark if the UDP packets exceed a certain size. We had cases where we had to first edit the PCAP in a raw text editor, seek to the problematic packet and then cut the rest of the trace off. Maybe someone can open a ticket with the Wireshark guys and let them know they should make the buffer bigger.

The PCAP recording goes beyond pure troubleshooting. Memory has become so cheap, that some clients record every call, regardless if they have trouble or not (sometimes you need to look up older calls). It can even be used to proof service level agreements. What is better than having the raw data for the call to prove that its quality was okay! I am not sure if there are tools available that can scan through the PCAP folders and automatically generate reports. Generating those reports can be done easily with a little delay. This would be a perfect job for a low-priority process on the system.

We are thinking about extending it even further. The first obvious move would be to have the PCAP accessible from the web interface, along with the call history; similar to the call recording. But we could also extend it further, so that the content of the call is not being recorded any more. In cases when only the call quality should be monitored, it can be an advantage not to record the voice itself. We could replace the voice parts in the RTP packets with zeros, so that there would be no problem having those packets archived. Of course, that would be a true waste of hard disk capacity. Originally I thought that RTCP-XR would do that job, but at least in our world we did not see that happen yet.