the tech that we love, will drive us mad

i have not posted much in the last week or so as i have been incredibly busy.  this week has been a good week just a tremendous amount to get done.  the strangest issue i have encountered this week was with the old and faithful source for strange problem generation.  the one and only, microsoft exchange.  i had a customer call and report that he could not get to the internet so he rebooted his servers one by one when he got to the exchange server and restated it, the internet came back up.  he called me and i found this to be quiet strange as would anyone else… i told him it would be right there and went to look at the problem.  when i arrived we verified that pulling the network cable would fix the problem.  i ran speedtest.net with the cable out and saw he had 3mbps down and 384 kbps up and a ping of around 70 ms.  when the cable was plugged into exchange he still had 3mbps down but only 2kbps up and and a ping of 999ms.  so i knew it was something trying to upload, but there was nothing visible in the smtp queue.  – a little back story, i had been at this customer’s site much earlier in the day to install antivirus on their servers and workstations since they had been experiencing problems with spyware pop-ups and the like, so my first though and his was to a virus. -back to the long and drawn out story at hand – we scanned the machine and found no virus files and still saw nothing in the smtp queue so i downloaded and installed wireshark (a great tool for analyzing network traffic).  i started a packet capture and then started the smtp service.  during the process of determining causation i had stopped the smtp service since i felt that was the most likely culprit for causing the outbound internet pipe to be full beyond capacity.  once wireshark was doing a capture i filtered the live capture for the smtp protocol and was rewarded with screen full after screen full of smtp packets.  looking at the ones getting the most hits i went to dnsstuff.com (worth every penny to subscribe) and did a reverse dns and found them to be valid email servers.  looking into the packets more i found that i saw the same username over and over again, so i had located the email bomber.  however, none of these messages were showing up in exchange and i needed to see them to delete them.  my next idea was to route all of the traffic to somewhere that did not exist.  i created a mail routing queue that went to a bogus address and i filtered all domains there with an asterisk.  then i restarted the microsoft exchange message routing service and all of the messages that had not made it out of the server yet, and those not showing up in the queue (but that were really in the queue) got diverted to this new queue, that went nowhere.  i did a select all and deleted all of these messages and the internet returned to normal and i reverse the flow of mail back to it ‘s normal path.  the mass of email we found in this junk box was astounding.  she had attempted to send an 8mb email to over 70 people, and it was not even in the relm of work related.  i love it the customer and i were kept after work until 9pm over someones personal emails done on a work system.  if i had a dollar for every time a personal issue affected a work system… well i would not be working anymore.  work computers are for work, home computers are for email forwards.  anyway i need to get down from this soapbox and get back to work.  the machine i am on is done with it’s windows updates and is beckoning me to restart it.

This entry was posted in Ramblings. Bookmark the permalink.

Comments are closed.