The ”30-day procurement challenge” – how US Citizen and Immigration service attracted more skilled vendors and improved quality (eng)

(This article is also available in swedish – read here.)

We took the opportunity to talk to Joshua Seckel, who has used Agile procurement within the US Government since 2013. Using these techniques, US Citizenship and Immigration Services has improved quality, drastically cut the speed of delivery and has attracted more skilled vendors to their programs. We wanted to learn how they made that change.

Hi Joshua, great pleasure to meet you. Tell us a little bit about yourself.

My background is in computer science, which I have complemented with an MBA. I’ve worked most of my career within the US State IT and the Federal IT. In 2013, I was asked to join USCIS (U.S. Citizenship and Immigration Services) as a division chief with the mission to modernize IT within USCIS. During that time, a large part of my work involved improving how we did procurement. During this process, we ran several experiments within USCIS and I also took part in the DHS Flash challenge. Today, I work as the Chief Solutions Architect at Sevatec.

Tell us about your procurement model at USCIS, how does it work?

I will share lessons learned from a very interesting project, something we called the ”30-day procurement challenge”. The idea was to cut the time from Request for Proposal (RFP) to Contract award down to 30 days.
The reasoning behind this was that our old government procurement processes took such a long time that when the procurement was complete, the business mission had essentially moved on.
The first time we ran the challenge, it took us 80 days from beginning to end. Not the 30 days we aimed for, but still way faster than the old approach. We also learned two important things:

1. Where our bottlenecks and waiting time was (so we could see where we needed to improve)
2. We got better quality with this approach. And the vast majority of our vendors liked it.

That’s interesting. Can you walk us through the key components?

The first key is to build close collaboration between Procurement, IT and your business. One example of this is to make sure your evaluation team has the skills to ask the right questions. In our case, that meant making sure that the evaluation team always had three skills: business understanding, procurement and IT.

The second key was to make sure that the vendor teams that participated did real demonstrations and not just hand over a written proposal. We did this through Code challenges. These were real coding challenges, where teams were asked to turn a small requirement into running software within 4 hours. This is where you can see and observe the real skills of the vendors.

The third and last key was to limit the number of vendor teams that could compete for the contract. In this specific instance, we limited the number of competing teams to 5. Remember, this was to meet the 30-day challenge. In other instances, the number of vendor teams can be higher. One extreme example is the DHS Flash procurement which involved 111 teams.

Which capabilities did you evaluate?

1. Can they actually get something built in production?
Regardless of what technique and process you’ve applied, having something to demonstrate and run at the end of the session is key. There were teams taking part in the coding challenge that didn’t keep this is in mind and as a result, had really thin demos.

2. Quality & craftsmanship
A great benefit of running a coding challenge is that you get to see and evaluate, with your own eyes, the craftsmanship and the best practices the providers use to deliver quality under time pressure. This matters because the quality of the provider’s craftsmanship would have a big impact on the end result in the real project.

As a part of the evaluation, we look at the code quality produced and the choice of craftsmanship practices used by the provider during the coding challenge. For example, if the provider makes use of pair programming and test driven development, the quality of the product is going to be higher. As a customer, you get to see the difference between teams that ”have read about it” and those who use it instinctively.

3. The collaboration
This turned out to be a big differentiator. We made sure we had a Product Owner (”PO” – a business representative) in the room. The question was, how did the vendor’s team make use of that? How do they elicit needs from the PO?  When the session began, we had a set of prepared user stories, which would be more than the team could ever complete within the given timeframe. They weren’t prioritized, but the Product Owner had a prepared prioritization in his back pocket.

The teams that didn’t do well in the collaboration either came in with a predefined notion of what needed to be built, and sought answers from the PO that confirmed this, or wasted time talking through and estimating all stories, before figuring out that it was too much.

The teams that did well asked the PO right off the bat what the underlying business need was in order to understand what mattered. Or they engaged the PO early in figuring out the priorities. Well-performing teams would also invest time in sketching the UI’s on paper and asking for feedback before implementing them.

We learned that even if the time frame was too short to see substantial differences in usability during the demos, we could evaluate the techniques they used in order to get good usability. The choice of techniques provided clear indications as to whether the team had the craftsmanship skills necessary to produce good usability.

How did you go about making sure that the vendors were evaluated consistently if the coding challenge took place many times?

We made sure that it was the same team evaluating all vendors. In the ”30-day challenge” we had one evaluation team that evaluated all vendors. In the DHS Flash challenge (which evaluated 111 vendor teams), we had 2 evaluation teams that evaluated them all. We also made sure that we clarified the capabilities we wanted to evaluate beforehand and made sure that the evaluation team had the relevant skills to do the assessment.

What was prepared before the code challenge? What did the vendors know about the project before choosing to participate?

In the 30-day challenge, the vendors knew the following:

• The business objective. This was prepared and communicated beforehand.
• How many teams are needed and for how long. In this case, we needed 1 -2 teams and this is what the vendors needed to commit to if they got the contract.
• The tech stack. In the case of USCIS, we do very little green field development, most new software is written on top of our existing tech stack. We have also made substantial efforts to make sure that we have a quick turnaround time from development to production. Therefore, in the case of the 30-day challenge, we selected the tech stack for the vendors.

There are other instances however where you’d benefit from leaving the choice of tech stack open. In the case of DHS Flash for example, we left the choice of tech stack open up to the vendors. This is because they would build software in a much more heterogeneous and fluid environment. In both cases, we used coding challenges to evaluate whether the vendors could demonstrate running software with quality.

What was the feedback from the vendors who took part in the new procurement process?

Most feedback was positive, vendors liked to be given the opportunity to show what they really could do. They also told us that if you do procurement well using a coding challenge, participating in procurement this way is actually cheaper than completing a written spec.

In the case of the DHS Flash procurement, which unfortunately got cancelled because of an administrative error, the vendors sent in an open letter to our Chief Procurement Officer saying ”we appreciate you doing procurement in this way, and we would like to continue doing it”.

So the vast majority of vendors do like this approach. The few vendors who aren’t overly fond of this approach are often great proposal writers who aren’t as strong in demonstrating quality delivery in accordance with what they propose.

You now have a couple of projects behind you that was procured using this approach. What are the key benefits from a customer’s perspective?

A few things:

1. To be able to see the work that gets produced. ”Show me, don’t tell me.”
2. We were able to identify areas in our own process which we needed to improve on. Where we had bottlenecks and waiting time in our procurement processes became transparent. By resolving these areas, we can overcome department silos and take the next step towards achieving 30-day procurement.
3. Lower barriers of entry. We get a higher level of interest from vendors. In the DHS Flash example, we were guessing that 50 teams would participate and we got 111!
4. Actually getting good vendors and teams on the contract. We can see that we are getting higher quality deliveries, and faster deliveries to boot.

How can you observe this?

I’ll share two cases in point:

• The first: Before, we had to train vendor teams in doing Agile. We don’t need to do that anymore.
• The second: The time between releases in the projects that was procured in this way was much shorter. A couple of examples: Our data analytics platform releases 50 times a year, which is roughly once a week. Our ELIS program, which involves 15-20 development teams, does 4 releases per week to production.

The lead time from prioritized idea to running in production is also shorter. The vast majority of our teams can take an idea to production within a month. And all of our teams could do it within 3 months.
This should be compared with the situation we had before (the norm within the US Government) where delivery happens only once a year.

What would you recommend others?

Just do it! Try it out on a small project right away. This allows you to learn how this works and will help you iron out glitches in your procurement processes. Secondly, make sure that you have the right people on the evaluation panel. People who understand IT and can discern good craftsmanship from the vendors. Thirdly, engage the business. Make the business or Product Owner a member of the evaluation team.
Finally, create good collaboration between Business, Procurement and IT early on. Having a daily standup meeting is good practice.

Thanks Joshua for taking the time to talk to us. We look forward to learning from your future experiments!