Troubleshooting is an unpleasant necessity when setting up and operating a PBX in the cloud or on customer premises. Our approach is to avoid trouble as much as possible by design, for example, by providing a NAT traversal solution for SIP endpoints. When it comes to trouble, however, it's important to have the necessary tools to find out, quickly, what's going on.

Troubleshooting SIP registrations is relatively simple. By checking out the SIP packets, most cases can be closed quickly. We added registration stability reporting practically from version 1: when an endpoint losses it registration or changes its registration address, the administrator gets an email. This makes it easy to locate buggy firewalls and unstable internet connections. We now have even the overview of how phones are registered in a domain, which very quickly shows where the problems are.

We have also added more and more features for troubleshooting the provisioning process. The PBX writes the generated files into the “generated” folder into the file system; here the administrator can take a look at what the phone has received for provisioning. The latest addition is a history file that shows when each file has been generated.

The most recent addition is the troubleshooting tool for calls. We have seen an increased need to troubleshoot media-related problems with the PBX: this is probably because more and more deployments are happening out there in the “cloud”, where quality of service isn't the same as in the LAN with a PSTN gateway. We have had RTCP-XR reports for many years, and those reports find their way into the MOS graph. They only provide a superficial overview on how users may perceive the overall quality, however, and in real-life PBX operations, the RTCP-XR reports aren'tt enough.

Our old approach to run Wireshark on the system was a good starting point. When you have hosted environments with hundreds of registrations and lots of calls, however, those PCAP files can get huge. What is even worse, though, is that finding the right call is like looking for a needle in a haystack.

In most cases, complaints about call quality affect a certain extension or trunk. Therefore, the PBX records the PCAP for calls based on the trunk or based on an extension. Each calls generates a separate PCAP file; if there are multiple calls that match the capturing criteria, the PBX will write several files accordingly.

And while we were on it, we also addressed another problem. Recording TLS and SRTP encrypted traffic with Wireshark is difficult by nature. Because the PBX has the security context for both the signaling and the media, it first decodes the traffic before sending it to the PCAP subsystem. Don’t be surprised if you find a SIP packet that was sent over TLS as a UDP packet in the trace - it's just to keep you informed as to what has been sent and received at what time. The RTP time-stamps are critical for the PCAP. In Linux, we are able to retrieve these time-stamps from the networking subsystem for received packets; for sent packets, the PBX takes the current time-stamp and uses it to add the packet to the PCAP file. This must also be done for the other operating systems that don't support retrieving time-stamps from the network subsystem (e.g., Windows). That said, the time-stamps should be good enough to troubleshoot visible and audible problems in the media stream.

The files are put into a directory that depends on the day. Inside the directory, the file name contains the time and the call port number, so it can be easily correlated to the other logs in the system.

The feature is available for all operating systems. This is another distinction from previous releases where PCAP was only available for Linux systems. It is now possible to generate SIP call traces for Windows, MAC, FreeBSD and, of course, the Linux flavors.

The new feature will be available in 5.1.3, with previews available in 5.1.2r.