How to Set Up Your Test Environment

As already explained in the article, "Understanding Computer Networks", computer networks are fairly complex, consisting of several different layers. To make things more complicated, these layers have a number of routine maintenance tasks, adding to the observed traffic. Therefore you need a test environment that is capable of distilling large amounts of data into the information you want.

Additionally, certain network activities only occur periodically. If the duration of your test is too short, you can miss them. If the duration of your test is unnecessarily long, then you can delay getting the results.

If your test environment is not set up properly, your results may be incomplete, or your test duration could be severely limited. This article lays out the fundamentals of setting up your test environment correctly.

Setting Up a Test Computer

Preventing automatic restarts

Most capture sessions for background traffic span a period of multiple days. Unless you are specifically testing system reboot scenarios, you won't want to get interrupted while you are conducting your testing. However, operating system updates may automatically restart your test computers, as well as the computers you are using for capture. This may cause your test session to be interrupted, and possibly even invalidate your results. It is far better to think about preventing such types of foreseeable system restarts, in advance; then you won’t get an unpleasant surprise when it is time to collect the logs from the test machine.

When you begin long haul zero exhaust testing, ensure that there are no pending updates and/or restarts. Any scheduled updates can be delivered later, and make sure that automatic restarts are disabled, too. On a stand-alone machine, you can update the registry to disable automatic restarts after the updates have been installed. In order to do this, run this command on an elevated command prompt:

reg add HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\WindowsUpdate\AU /v NoAutoRebootWithLoggedOnUsers /t REG_DWORD /d 1 /f

Afterwards, your registry should look like this:
Disabling Windows Automatic Restart after Updates

On a domain joined machine, you can use this Group Policy object.

Computer Configuration / Administrative Templates / Windows Components / Windows Update

"Enable" the following policy:

No auto-restart with logged on users for scheduled automatic updates installations

Flushing DNS Cache

If you want to know the destination of your traffic, you will need to know the domain names for the IP addresses in your captures. After all, www.example.com is much more descriptive than "93.184.216.34". Sometimes it may be possible to make a, so-called, reverse DNS lookup to find out the name that is currently assigned to that IP address. However this method is error prone because IP address assignments change frequently. Additionally, reverse DNS lookups fail quite often.

Thankfully, there is an easier and much more accurate way. When your computer is asked to connect to a certain domain name, it finds the corresponding IP address using Domain Name Service (DNS) servers. These requests are also network queries that will be recorded in your capture sessions. Smart tools, such as Traffic Hound, can parse this traffic to learn which IP address belongs to which domain name. The biggest advantage of this approach is that it is very accurate, as it returns the IP address - domain name pair, at that point in the time.

Since DNS queries are relatively costly, operating systems tend to cache the results for a certain period of time. If a result can be found, simply, in the cache, there won't be a corresponding DNS query, and therefore it won't be caught in the capture files, which will make it much harder to analyze IP addresses later on. So, before starting your capture session, you should "flush" the DNS cache to force your operating system to make fresh queries that will be picked up in the capture files.

To flush the DNS cache, run the following command on the command prompt. It is important that you do this just before starting your testing.

ipconfig /flushdns

Where to Capture

You have essentially four options:

  1. Capture on the test machine itself
  2. Test machine runs on a Virtual Machine (VM), with capture on the host machine
  3. Capture on the network
  4. Capture at the gateway

1. Capturing on the test machine itself

This is by far the easiest method. All you have to do is install your favorite capture software on the test machine. This could be the right choice if you want to verify the effect of a setting, or make sure a specific application is not leaking any traffic. However, this method is not recommended for whole-machine operating system level testing, for a number of reasons:

  • The capture software may generate its own traffic while checking for updates, etc. This can become confusing when analyzing the results.
  • Running testing on the same machine can mean there may be may be resource contention which can affect the behavior of the machine. For example, the capture software may retain captured packets in the memory as opposed to writing them out to a hard disk. After a long test session, the capture software's memory use may grow significantly, causing resource pressure on the rest of the system. This could mean that because of these temporary resource constraints, certain tasks might be postponed until after you close the test software. The result of this could be that you possibly miss important traffic, related to those tasks.
  • Some systems can detect when there is capture software active and will take evasive action to become invisible. This is especially the case if you are testing for the presence of malicious software (malwares, viruses, etc.).

Although it's not ideal, this method may be your only option, if you are testing in a cloud environment where you typically don't have access to the host computers. You may be able, however, to create virtual networks in the cloud. If that's possible, you can capture traffic at the designated gateway. Please see item #4.

2. Test machine runs on a virtual machine (VM), with capture on the host machine

This is the most highly recommended method. It has a number of advantages and no major downsides.

  • Although both the capture software and test machine are running on the same physical machine, they are relatively well isolated from each other. The capture software runs in its own isolated environment, minimizing the chance of any harmful interference between the capture system and the test system(s).
  • The test machine doesn't know it is being monitored; therefore, any malicious software can't hide itself.
  • As is the case for virtualization in general, this is a more efficient use of hardware resources, and it allows you to conduct more tests simultaneously.

Setting up a virtual machine properly is not without some complexities. We will point out the most important aspects.

Internet Connectivity

Make sure that your VM can connect to Internet. Open your favorite browser and try to reach a popular website. Does it work? If it doesn't, you may need to create a virtual switch and connect your VM to it. Here is how to do it this in a Hyper-V environment:

  1. Make sure that your Hyper-V server is turned off. Do not use the "Turn Off" command, as that is more or less the same things as virtually pulling the plug on your VM; not giving it a chance to shut down gracefully. Shut down your computer from the guest operating system (OS), or by using the Hyper-V shutdown command (if available).
  2. Create a "Virtual Switch" for your VMs. In the "Actions" menu, click on "Virtual Switch Manager". If you don't have any switches yet, create an "External" virtual switch. Name it "vInternet". Make sure the selected connection type is still "External Network" and ensure that the "Allow management operating system to share this network adapter" checkbox is selected.
    Creating an external switch in Hyper-V
    Creating an external switch in Hyper-V
  3. Now go to settings. Hyper-V offers you no less than three different ways of accessing the settings dialog. You can use application menu; or context menu, by right clicking on your VMs name or the "Actions" menu on the right.
  4. Connect your VM to "vInternet".
    Choosing Hyper-V network adapter

Resource Allocation

When applications and operating systems are resource-starved, they stop their normal activities. To prevent such an artificial network silence, you need to make sure that your test machines are well taken care of. Make sure that you allocate enough resources to your test machine when creating a virtual machine. We will describe how to ensure the necessary settings for the Microsoft Windows Hyper-V server, here.

  1. Make sure that your Hyper-V server is turned off. Do not use the "Turn Off" command, as that is more or less the same things as virtually pulling the plug on your VM; not giving it a chance to shut down gracefully. Shut down your computer from the guest operating system (OS), or by using the Hyper-V shutdown command (if available).
  2. Now, go to settings. Hyper-V offers you no less than three different ways of accessing the settings dialog. You can use the application menu; or the context menu, by right clicking on your VM's name or the "Actions" menu on the right.
  3. In the settings dialog, give it the right to use to as many cores as you can afford; even all of the cores that are available on the system, if possible. Don't worry; this won't starve your host computer. By default, they are not exclusively dedicated to this particular test machine unless you go ahead and change the setting for "Virtual machine reserver (percentage)". If you are worried about system performance, consider throttling the cores down to 50 or even lower. That way, your test machine will have multiple processing cores available to it, but you will also be protecting the host machine's performance.
    Hyper-V settings for CPU allocation
  4. Allocate no less than 2GB of memory to the virtual machine. In our example, the host machine has 32 GB of RAM available. Since we can afford it, we allocate 4GB (4096 MB). Please note the checked "Enable Dynamic Memory". Similar to CPU, this feature allows for the most efficient use of available host memory. If the VM needs memory to perform complex tasks (which may cause network exhaust), it can get it. If not, the host's memory won't be wasted, unnecessarily.
    Hyper-V settings for memory allocation

3. Capture on the network

The biggest advantage of this, over #2, is that this allows test machines to run on real hardware (vs. on a virtual machine). Most operating systems are "enlightened" when running in a virtual machine environment, which means that they know that they are on a virtual machine and act slightly differently, so that their performance can be optimized. This behavior may alter their network traffic patterns, thus reducing the validity of your tests.

You will need special hardware to do this type of testing. As you may remember from the article, "Understanding Computer Networks", network switches only forward packets to ports which the destination host is connected to.
Isolated Switch Ports

There are several ways of tapping into this restricted channel. For example, more advanced switches support, so-called, port monitoring, allowing you to configure them in such a way that they forward network packets to your choice of other ports so that they can be monitored there. This is also our preferred method.
Port Forwarding

Alternatively, you can put a "hub" between the test computer and the network switch. Hubs forward all traffic to all ports. Normally this makes them very inefficient, however, in this particular case this allows us to capture traffic originating from one port on all other ports. Do note, however, that not only our capture machine will see test machine's traffic, but unfortunately the test machine will also see the traffic from our capture machine. This is OK in most cases, though. This extra traffic is local area traffic and shouldn't cause any Internet-bound network exhaust. This is an acceptable alternative to switch port forwarding.
Port Monitoring with a Hub

The final option is to buy a dedicated hardware sniffer. These devices have In and Out ports that let the traffic go through while monitoring it at the same time. These devices are becoming a rarity these days however, as most capture needs can easily be met using one of above set-ups and a laptop.

4. Capture at the gateway

Similar to #3, this allows test machines to run on real hardware. However, there are some considerable drawbacks:

  • Gateways impact your entire network. While doing the set-up, you can accidentally interrupt the network connectivity for your whole organization.
  • Since gateways are the choke points for a large set of network devices, they receive and process large amounts of data. Under such a heavy load, the capture software may be forced to ignore packets, causing you to possibly miss important traffic. In addition, unless you craft your capture filters carefully, your log files will grow very large relatively quickly. Finally, all of this extra processing may slow down your organization's Internet access.
  • You can assign a dedicated gateway to your test machine, leaving your organization's main gateway alone; however the correct set-up of such an environment is relatively difficult.

Because of these drawbacks, this method is generally not recommended. Possibly the biggest exception to this is testing on cloud environments, where you typically don't have access to the underlying host operating system or network equipment. If your cloud provider offers virtual networking features, you can route traffic from your VMs to a designated gateway, capturing traffic there. We will cover this in a future article.

Capture Tool Choice

When doing zero exhaust testing, we want our tools to be reliable. We've had cases where our test tool crashed after only a few days into the testing period and we lost all the logs after that point. Imagine our big disappointment when we tried to collect the logs after two weeks of testing!

Additionally, we want our tools to be able to monitor all traffic but to be configurable enough to only capture what is relevant to our purposes. This optimizes resource use during testing, as well as reducing the time needed to analyze the results.

We recommend three tools for capturing outbound connections

  1. Traffic Hound
  2. Firewall Logs
  3. WireShark

1. Traffic Hound

Traffic Hound is a custom-made tool, dedicated to zero exhaust testing. Since it is a purpose-built tool, it has many advantages over other general purpose capture tools:

  • It is pre-configured to capture only the most interesting traffic. For instance, it automatically ignores local area network (LAN) traffic that is needed for your network's proper operation. Traffic Hound also ignores traffic that is addressed to local area network machines such as your domain controllers, etc.
  • Since it has a very specific focus, it is very resource efficient, allowing long test sessions in the busiest of networks.
  • It automatically parses DNS packets captured in the traffic, giving you a definitive IP address to map domain names, and thus eliminating the guesswork from figuring out where your data really goes.
  • Its results are very easy to interpret, removing the chance of misinterpretations.
  • It is geo-aware, helping you understand where your traffic actually goes.

2. Firewall Logs

This is the most "lean and mean" approach. Firewall logs contain high-level information about allowed connections. Usually, it is possible to extract destination address, protocol type and time of transmission. When you are done, you can import the logs into a spreadsheet application such as Microsoft Excel and analyze them further. It is preferable, however, to use Traffic Hound to parse and analyze Microsoft Windows Firewall logs.

3. Wireshark

Wireshark is possibly the most popular network capture software. It has a wealth of features, but it is still relatively easy to use. Wireshark's biggest advantage is that it is supported on many different platforms, including Windows, MacOS and popular Linux distros.

Since Wireshark is a general purpose tool, you will need to make a few configuration settings to ensure optimum performance for zero exhaust testing. First go to the "Capture" menu item, in the application menu:
Wireshark Start Capture Command Menu Location
In the Input section, choose your network adapter. Add the following filter to eliminate all link-layer traffic, which is usually very noisy and uninteresting as it never leaves your network:

IP or IP6

Wireshark Start Capture Input Settings

Next, in the output section, choose your output folder. Make sure that you select the "pcap" file format to ensure Traffic Hound compatibility. Files will be broken into smaller chunks to maximize their protection in case of data corruption.

Wireshark Start Capture Output Settings

In the options section, disable network name resolutions (because they are not particularly accurate) and because we will do our own analysis later. It is possible that they may cause their own traffic which would interfere with our analysis.

Wireshark Start Capture Options

Below is the command line that achieves the same settings using command line tool "tshark":

"c:\Program Files\Wireshark\tshark.exe" -f "ip or ip6" -n -F pcap -b duration:21600 -w %USERPROFILE%\Desktop\capture.pcap

Here, "f" is the capture filter. "k" instructs the capture to begin immediately. "n" disables name resolutions. Wireshark name resolutions cause extra traffic and they aren't always accurate. "b" breaks the capture files into smaller chunks. In this case, a new capture file will be created every six hours. This will ensure maximum data protection in case of a crash or corruption. "w" defines the output location, in this case, the desktop of the user who ran the command.

Did you forget to configure Wireshark to capture in "pcap" format? No problem, you can bulk covert them using "editcap" that comes with Wireshark:

for %I in ("C:\Location Of Your Capture Files\*.pcapng") do (editcap -F pcap "%~fI" "%~dI%~pI%~nI.pcap")

Wireshark is "free software"; you can download it without paying any license fee. The license under which Wireshark is issued is the GNU General Public License version 2.

Wireshark Download Link

The good news is that Traffic Hound is Wireshark compatible. You can feed your Wireshark logs into Traffic Hound, making zero exhaust privacy analysis much easier. When capturing with Wireshark, make sure you use the "pcap" format, to ensure compatibility with Traffic Hound.

Not Recommended

We don't recommend these tools as a primary choice:

  • Fiddler by Telerik. Although it's a great web development / diagnosis tool, it is not suitable for general zero exhaust testing because it can only capture HTTP traffic. Fiddler will miss any other TCP, UDP or ICMP traffic.
  • Development on Microsoft NetMon stopped a long time ago, and it is currently out of support.
  • Microsoft Message Analyzer, the replacement for Microsoft NetMon, is a great tool with many features that no other tool has, but it is notoriously difficult to set up correctly. Since it doesn't provide any material advantages over other tools, we think that it is not worth the effort.

Capture Duration

When evaluating operating systems for applications in sensitive environments such as in government and enterprise installations, we recommend long term testing. Below is our advice for timings:

  • 8 days - Some activities in operating system run on a recurring schedule. Good examples of such periodic traffic are checking for updates and sending telemetry. We believe that 8 days, slightly longer than a week, gives a good chance of capturing most of such periodic traffic.
  • 32 days - Some activities in operating system are spread out over longer stretches of time. These activities are either costly or they don't need to be checked often. For example, the verification and renewal of licenses for purchased apps doesn't need to be done often. Additionally, downloading OS updates is carried out infrequently because software vendors try to group their updates together in order to reduce the disruption that the updates cause. Doing long haul testing for 32 days ensures that you will catch all daily, weekly and monthly traffic.

Comments? Questions?

We want to hear from you, please let us know if you have any comments or questions