QSA Vendor Selection – Points of Consideration

Earlier this year I lead a QSA selection activity for a large PCI-related program I am the security lead for. Thanks to an email conversation this morning – with a friend who is crafting a QSA-related RFP – I want to share some points of consideration that I shared with her.

1.    Carefully craft your RFP. Know what you want to get out of the engagement. Thus, when you read the responses – you may be able to quickly separate QSAs that did not take the time to tailor their response (and thus did not understand the engagement as a whole) from those that actually read it, understand your needs and want the business. In my case, before we allowed vendors to respond – we had a huge conference call. I allowed all the vendors to ask a few questions. In interesting observation from this call was that after the first four (of 12) vendors asked questions – there were no more questions. I guess they tend to ask the same questions. In addition, I think the conference call scared off some vendors from actually responding. They realized that we understood PCI-DSS and they were not going to be able to sell a shoddy engagement.

2.    Specify your minimum experience expectations for vendor personnel that will be doing the actual work. The PCI SSC outlines minimum requirements. I tend to have higher expectations and have no problem forcing my expectations on vendors. I want a QSA assessor that has between 5-7 years of “information security” – not auditing – experience. In addition, I want someone that has a certification from the Society of Payment Security Professionals. Finally, I want a QSA assessor that has been doing PCI-related assessments / consulting for at least two years. Some QSAs will balk at these experience expectations – but again, it is my engagement and my choice and I will validate that they are meeting my experience expectations.

3.    Request Resumes. Dictate that the QSA vendor provides resumes from the pool of individuals that could be performing the work. There will always be a chance they do a bait and switch on you – that is a different problem.

4.    Interview the person(s) that the vendor foresees performing the engagement. The sales / account manager may also balk at this – which if they do – that should be a red flag. The serious QSA vendors should have no problem doing this. And guess what – if the vendor pulls a bait and switch on you after the work has been awarded– demand that you interview the replacement before the actual work begins. You need to be comfortable with the QSA assessor.

5.    Validate Estimates. Make sure that the estimates the QSA vendor provides are realistic; this is a shared responsibility between the merchant and the QSA vendor. I cannot underscore this enough. Some vendors will low-ball their estimates for the hours needed to make themselves more appealing from a cost perspective or simply to provide a less then complete assessment. Each environment is unique so assessment times will vary. Regardless have another set of eyes review the estimates to make sure they are fairly realistic. Also, double check the hours needed for documentation. I am a big proponent of having ample documentation time. However, when vendors abuse the use of templates and do not take the time to do real, comprehensive documentation – that makes me really upset. This is probably a separate blog-post topic.

6.    References. Have the QSA vendor provide references. Again, they may balk or drag their feet on this. Also, keep in mind that they will not provide references from unhappy customers. The way around this is to make sure you ask questions to the happy customers that give insight to things like timeliness, quality, business acumen, and skill sets of the QSA assessors themselves. Also, get references from clients of the QSA vendor that are in the same industry and the same merchant level as you (this should already be a requirement for in your RFP; that the QSA vendor has performed QSA-related work in your industry and at your merchant level).

7.    QSA Feedback Forms. Make it known that you fully intent to provide the PCI SSC with a QSA Feedback form after the engagement with the QSA vendor. The form can be found here and can be submitted by the QSA vendor client directly to the PCI SSC. The QSA I chose never gave me a feedback form and I am debating whether or not I want to share my feedback – that I have already shared with the vendor – with the PCI SSC directly.

8.    Be familiar with the QSA Agreements and QSA Requirements. You should expect to get responses from QSA vendors that are probably in violation of these two documents. I certainly did and guess what – those QSA vendors – yes, more then one – were removed from my consideration. You can find these documents here, here and here.

In summary, one way that the PCI SSC and QSA market can get better is by merchants better educating themselves on PCI-DSS and the QSA market. Merchants need to understand that they have resources to make sound QSA selection decisions as well feedback loops to help the PCI SSC perform some QA on the QSA vendors community as a whole.

Advertisements

12 Responses to QSA Vendor Selection – Points of Consideration

  1. This advice is a bit egregious.

    Number Six I completely agree with, but the rest is excessive and unnecessary.

    Spend money and time on compliance readiness with an external trusted adviser, such as a strategy consulting company specializing in risk management, with strong experience and exposure to both network and app penetration-testing. Risk management is so much more important than any single compliance standard (or even all of them combined).

  2. Chris Hayes says:

    @Andre – I respectfully disagree on the word egregious. The reality is that some merchants are required to use QSA vendors. In addition, when merchants that are only required to do SAQs but are looking for subject matter expertise on PCI-DSS – QSA vendors are the most appropriate choice in the eyes of most payment processors – especially when compensating controls need to be used.

    I think you are reading too much into this post. In addition, if you have read any of my previous posts – it should be apparent that I clearly differentiate between being compliant, being secure, and managing risk. Regardless, thank you for taking a few minutes to read the post.

  3. […] replace QSA with penetration test, consulting gig, etc. These are great tips for all RFP processes. QSA Vendor Selection – Points of Consideration << Risktical Ramblings Tags: ( rfp […]

  4. cmlh says:

    @Chris Hayes

    In relation to 2.

    Are you referring to the CPISA?

    Have you encountered anyone with both the CPISA and CPISM?

    Also, what has been you experience with QSA outside of the USA?

  5. @Chris Hayes:
    You are right that I am reading too much into the post. That’s what I do.

    This is a really good post, mind you, otherwise I wouldn’t have commented!

    However, I just think that your #6 should be #1 and the others should be like… footnotes or something ;>

    To continue the good conversation, I wanted to comment further on one point you made, “QSA vendors are the most appropriate choice in the eyes of most payment processors”.

    This is an interesting insight because what if QSACs are not the most appropriate choice to help with SAQs, compensating/alternate controls, or planned/future Merchant Level changes? My strawman argument here is that QSACs are notoriously low quality and lack proper risk management experience, as well as deep technical experience.

    Also — I have some questions based on this comment. Assume you are right and that QSACs are the most appropriate choice to go to for compliance readiness projects for companies handling cardholder data.

    Who is the next most appropriate choice? What kinds of non-QSAC security consulting/services shops or security vendors should prospecting organizations avoid?

    I mean, it doesn’t make any sense to hire out a QSA to do work that isn’t required by a QSA, right? Or are you suggesting that prospecting organizations instead use non-QSA people from a QSAC for all compliance readiness work? Again, this brings me back to my point that QSAC != quality.

  6. Chris Hayes says:

    @cmlh – My reference to the Society of Payment Professionals was primarily in the context of the CPISA certification. From what I understand, the CPISA exam by far more comprehensive in terms of what it covers. QSA + CPISA + a good interview hopefully results in a half decent QSA assessor. Regarding QSAs outside the United States – no idea – sorry. I would hook up with @sfoak on Twitter – he can probably point you in the right direction. Thanks for the comment!

    @Andre – Thanks for the follow-up comment. The reason I stated that the QSAC is the most appropriate is based off my experience of payment processors deferring to the QSAC as to whether or not a compensating control is adequate or for interpreting various PCI-DSS language. Personally, I think it is a cop out on the payment processor side but I have witnessed these conversations. So, you wind up in a situation where you have a sub par QSAC stating one thing, a merchant stating something not in alignment with the QSAC, and the payment processor siding with the QSAC because they are the independent PCI qualified assessor. This is a liability shift-play on the side of the processors.

    Without going into details, the reason my employer recently reached out to a QSAC was to get an objective third party review of a solution we are building. Once you become a level one merchant, you have to use a QSAC. Thus in scenarios where you are a level 2 merchant, almost a level 1 merchant, and spending a lot of money on a solution – you want to make sure that when you do become level 1 your solution is tight – otherwise it is a waste of money.

    We are more in agreement then we are in disagreement. Some of my past PCI / QSA posts underscore my skeptcism towards QSAs- but reiterate merchants taking more responsibility for PCI-DSS. At the end of the day, I have to be the more knowledgable about PCI-DSS then any QSAC we hire; unforutnately – they carry a lot more weight in their opinion.

    Thanks again for the comments Andre. Glad to see that a fellow OWASP leader cares about PCI and risk topics.

  7. Alex says:

    @Andre

    “Spend money and time on compliance readiness with an external trusted adviser, such as a strategy consulting company specializing in risk management, with strong experience and exposure to both network and app penetration-testing. Risk management is so much more important than any single compliance standard (or even all of them combined).”

    With all due respect, I humbly submit that there are very, very few information security focused “risk management” consultancies that could teach Chris (and his organization) much of anything.

  8. cmlh says:

    @Chris Hayes

    I have known @sfoak for sometime and he and @chipmonkey allowed me to stay with them when I arrived in the USA in September 2008 to present at the OWASP USA Conference and ToorCon prior to departing to Toronto, Canada for SecTor 2008.

    I suspect internationally there would be not much difference as @sfoak also conducted the QSA training internationally (and I believe he also taught the SPSP Training but I could be incorrect about this).

  9. Trebuchet says:

    Chris … how much are you being paid to market their Society of Payment Security Professionals certification?

    I probably have more experience than a vast majority of the QSAs currently in the market. Over the past 7 years I have not only conducted countless audits, but have leaded the two largest remediation efforts currently on record for the PCI Industry. I also, have an extensive background in IT Security, Forensics and Intrusion Analysis, and carry the commensurate certifications as well.

    In my opinion, The Society of Payment Security Professionals is just another new certification from a small group of well meaning individuals who are trying to make a name for themselves as the “defacto” standard-gurus in the PCI Industry. I also believe it is against the guidance of the PCI SSC and could place needless legal exposure to a well intended effort.

    Also, as a corporate official, you should be coached as to what certifications you “recognize” and make sure your corporate legal staff is also in alignment with such attestations.

    More importantly, from my experience, I have found that an IT Security and Audit background is a good start … but a QSAs metal is forged in remediation efforts not in the “wham-bam-thank-you-ma’am” of audit.

    Good luck in your search.

  10. Chris Hayes says:

    @Trebuchet – Thanks for the comment and sharing some insight. And no, I am not being paid by any one for my comments on the SPSP.

    But hey – thanks for the chuckle on my second day of vacation. Off to the beach…

  11. Dear Chris!

    I kindly ask your permission to publish this article on under column Knowledge base on my company Web site, also we ‘re going to translate this article to ukrainian and russian languages and put it to respective chapters of the site.

    Our company is focused on IT audit and IRM functions, operating in the Ukraine.

  12. Chris Hayes says:

    @Volodymyr – Please accept my apologies for taking so long to respond. You are more then welcome to publish the “QSA Vendor Selection” article on your company website. Thanks for asking!

%d bloggers like this: