Clear Nullmailer Spool - Resolving Endless DNS Requests

Today I was looking around for what I thought was a routing or firewalling problem (there was a virtual machine that couldn't access to Internet). Playing with the firewall, activating rules, permitting everything and logging everything, I finally found and resolved the problem. Before reverting to the base situation and just editing the needed rule, I took a look at the logs.

What I saw were 20-30 DNS requests per SECOND from a single Ubuntu 12.04 machine. WTF?? That's a very strange behaviour. I started looking around for what was causing this problem.

QUICK ANSWER
In my situation, what I had was a misconfigured mail client (nullmailer, obviously). There was configured a non-existing internal SMTP server (let's say smtp.server.internal), so it requested the DNS server to resolve the name (once for each mail!!!), which didn't exist,  so it couldn't send any mail. So my nagios (Nagios Website) alerting mails were just accumulating in its queue, making the problem worse with every mail.

What I had to do was first to clear the spool (all mails were simply alerts, so I could remove them without any problems):
rm /var/spool/nullmailer/queue/*

Then I had to configure the right SMTP server for nullmailer in /etc/nullmailer/remotes, replacing smtp.server.internal with, for example smtp.gmail.com and the right credentials, and everything was ok.

 LONG ANSWER

PACKET SNIFFING
First, I took a Packet Capture from Pfsense firewall (Pfsense Website). That's a cute firewall, in my opinion, but that's not the argument today.
Packet Capture is a packet analyzer (or packet sniffer); that's to say, a tool that receives every TCP packet on a machine (you know about TCP protocol? Wikipedia TCP for a more rigorous explanation, What is a packet? for a simpler explanation). Then, you can filter what is captured to show only packets on a certain interface, or coming from a specific IP, or almost everything else you may want.
Whit this tool, I captured all the packets from the interface on which the Ubuntu machine was placed, from the Ubuntu machine IP address. Thenp, Pfsense has another cute feature: Download Capture. You can download the results of the capture in a .cap file, to be open with a specific program: WireShark (WireShark Website). Wireshark is a great tool for any system administrator; sooner or later you're gonna need it. Incidentally, it can be used not only as viewer but also as Packet Sniffer. If you can't use a graphical instrument to sniff, you can use tcpdump (TCPDump Website) on Linux. You can filter data almost in the same way, and output the results in the command line or in a .cap file. Note: to capture all the packet, and not just the first part containing addresses and routing protocols, you need to use a command like this:
tcpdump -i <interfacename> -s0 -w <filepath>
where -i is the interface, -s0 captures all the packets, -w writes the output to the indicated file.
As a reference, I used TCPDump Tutorial and Capturing with tcpdump for viewing with Wireshark.

Well, coming back to my story, I then looked at the captured packet with using Wireshark. This is what I found
General output of my packet capture.
Well, at least I had a general indication of my problem. My Ubuntu server was trying to send a lot of mail through "mail.dr", and the DNS server was responding that the name requested didn't exist. It is obviously a misconfiguration problem.

UBUNTU SERVER LOG
Well, if I have some kind of configuration problem, let's look at the logs of the machine! I opened /var/log/mail.err (but depending on your OS, maybe you will find something on syslog) and this is what I found:
nullmailer? That's interesting!
Another clue! nullmailer: what the hell is this? A little search showed that nullmailer is a simple mail transport agent, a program that sends mails to an SMTP server (NullMailer Homepage). I had in fact a program that should have warned me via mail of the results of a daily operation now that I remember it, but never did. Well, and now? 

 CLEAR THE SPOOL
As one of my university professors always said, a telecommunication engeneer has two parents: Mother Google and Father Shannon (fs > 2fm - oh, sorry, it's almost automatic for me). It's time to call for Mother Google to help us. And guess? It did. 
This post showed me the light: Endless entries of nullmailer. The program trying to send is called nullmailer, which put the mails in the folder /var/spool/nullmailer/queue, waiting to be sent. Then periodically nullmailer tried to send all the mails using a non-existent SMTP server. When I found out this, there were more than 3k (luckily unimportant) mails to be processed. To temporarly solve the problem, I had to delete all mails in /var/spool/nullmailer/queue: 
rm /var/spool/nullmailer/queue/*

 Pfiu! Let's check the firewall, to see if it is effectively solved this part of the problemi. No more DNS requests. Well, yeah! I rock! ;)

 Obviously, now I have to install and properly configure a mail sender now. Work never stops! But maybe I'll tell you about this another time.

 RELATED POSTS


Labels: ,