Wake Up Call to Software Vendors

While working on deploying an update to a software application by one of the vendors I deal with on a regular basis, I was given a document titled "Pre-Install Handout".  (Well, I'm not really that crazy about handouts.  Most of the time I believe in earning my keep. Anyway...)  At the top of page three, there is a big yellow triangle with a bold black exclamation mark next to the following text:

"Domains: If the [server] machine is to be placed on the [customer] domain, [vendor] must have complete administrative rights to that machine.
 
There should be absolutely no security restrictions on making changes to that machine. If the domain security interferes with the [server] machines operations; it will have to be removed from the domain."

Dear vendor, if you can't get your software to work with the security settings of our environment, then it is probably time for me to start looking for a new vendor.

This is the same vendor that a couple of months earlier, I sent a request asking for a copy of their SSAE-16 reports and financial statements so that I could include the documentation in my vendor management program file.  The response I got back:

"Though [Vendor] does have products that fall under the requirements of a SAS70 audit, because both [Product A] and [Product B] are considered in-house products for our customers (no data is stored on our servers here), it does not apply in this instance.

I have attached an FAQ that further addresses your request. As a point of mention, both the [Product A] and [Product B] divisions of [Vendor] share a facility with the [Product C] division and though we are separate products, we are bound by the same facility and information security management as addressed in that product’s SAS70. Let me know if I can provide any additional assistance.

The SAS70 report appears to now be in the new SSAE-16 reports, but again, the [Product A] and [Product B] departments, this doesn’t apply." 

A couple of weeks after getting this response, I had a support ticket open with this vendor.  The same support technician that sent me the response above, proceeded to download a copy of the application database from my server to their workstation to "test some things out".  The vendor did not request permission before making a copy of the data that I am responsible for, and didn't admit to doing so until after I questioned them about it.  I then asked for a copy of their SSAE-16 reports (which they obviously don't have).  Finally, the vendor sent me an updated privacy statement that included coverage of customer data within the vendor's environment.  While not a perfect solution, it should hold up in court if they have a data breach.  But at that point all of the data I'm responsible for is probably already on pastebin.

So back to the software upgrade deployment I mentioned earlier.  When I sent the paperwork in to the vendor for the service request to have them install the upgrade (and no, they don't allow anyone else to do the installation), I explicitly stated that they are not to make any changes to the OS, firewall settings, install third party tools, etc. unless they first got permission to do so.  I also restated this to the technician (different than the one above) working on the install and got an affirmative response that they agreed to these rules of engagement.  So guess what?  I remote into the system they are installing the upgrade to and see the vendor making changes to the host firewall (as in completely turning it off).  #FAIL!  I then terminated the vendor's connection, turned the firewall back on, and called to see whether this technician suffers from short term memory loss.

Dear vendor, when are you going to wake up and treat your customers with a little bit of decency.  It is not my goal in life to catch you doing stupid things.  I really don't need the extra stress of worrying about what you might do to compromise the security of my data.  I have enough bad guys out there trying to break into my systems without you handing them the keys to your backdoor.

A word of advice to sysadmins and business folks that deal with vendors:

I doubt I'm alone in this experience.  I suspect things like this happen everyday to systems in your environment the same as they do in mine.  Do you have policies in place to handle these situations?  Are the people on your team trained to watch out for this type of negligence on the part of the vendor?

One of the basic concepts of data security is the principle of least privilege.  This should be applied to vendors as much as anyone.  Vendors don't usually have malicious intent, but in general they also have much less skin in the game when it comes to protecting your data.  Monitor their activity.  Watch what they are doing when they are in your system.  Revoke their access unless you have a real need for them to be in your systems.