Judging a hackathon with the Volley decision science platform
Using collaborative decision optimization (TM) with qualitative criteria to select the winning teams
Recently a hackathon was hosted at the University of Colorado Boulder — Leeds School of Business. Not just any hackathon (not that any hackathon is “just a hackathon”… every event is interesting and unique!). Instead, this was the first-ever CU Low-Code Hackathon and it was a huge success.
Over $10,000 of prize money was awarded to the non-profit partners and winning hack teams. Certification vouchers and vendor merch were plentiful. And there was free food… which drives attendance and fuels competition.
Dozens of students majoring in Business, Engineering, Liberal Arts, and Interdisciplinary Studies came together during a cold weekend in Colorado to build low-code applications for non-profits in the Boulder / Denver community… joined by dozens of industry mentors who graciously volunteered to guide their progress through the 60 hour “sprint” and judge their final results.
Each hack team was assigned to a specific use case representing one of the non-profit partner’s actual needs, spanning Fundraising, Volunteer coordination, Campaign tracking, and Resource assignments. The teams worked through an agile lifecycle to understand the client’s requirements, conduct Design Thinking exploration, and then rapidly build, test, and launch their application. In order to deliver the greatest amount of value for the non-profit partners, no two hack teams built the same solution. That said, it made the judging of the winners a much more complex process… requiring that the criteria and the decision process needed to be very robust.
The criteria for judging were based on four dimensions:
- Creativity — how “out of the box” the hack team was in solving the need and extent to which existing tools were reused & extended
- Design Thinking — the extent to which observation and empathy were employed in the process of understanding user needs through an interactive ideation & prototyping process
- Execution — the level of professionalism and proficiency demonstrated in the application developed by the hack team
- Feasibility & Scalability — how likely is it that the partner can easily begin using the application, and how significantly the application can grow to meet the partner’s growing demands with reasonable cost impacts
Each of these dimensions clearly ties to the value and impact that an application can yield. You may notice that each of these dimensions is qualitative (no common way of quantifying since there is no unit of measure) and subjective (no widely agreed-upon definition of excellence). Hmmm.
The aforementioned guidance on judging dimensions was provided to help align contestants and judges, but there was still room for some interpretation. And if a rubric were created, what would “very creative” mean… or how would “decent execution” be quantified? What would a score of 7 actually represent for Design Thinking?
Foreseeing this complexity… which is very similar to the material decisions that all organizations make all the time…. the event was planned and run with significant thought to Decision Science and Decision Quality. Due to the nature of the qualitative and subjective criteria, and each team’s submission targeting a different client use case, a methodology and system were clearly needed to manage the process and determine the outcomes. Add to that the rapid pace of the finalist selections, presentations, and award judging, a Decision Science platform was used to select the winners… with speed and confidence. It was the perfect use case to utilize the sophisticated technique called Analytic Hierarchy Process (AHP), a long-standing and well-proven statistical approach that is based on conjoint analysis and multi-criteria decision analysis. AHP is very common in the fields of academic research and theoretical pursuits, but almost unheard of and even less commonly used in organizations.
If only there was a secure web-based platform that supports AHP analysis and is easy for anyone to use…
And that was: Volley, the collaborative decision optimization platform. Powering the hackathon judging meant that Volley would be utilized to provide two critical capabilities:
— AHP analysis via the Volley Decision Engine (TM)
— Decision debrief via the Volley Outcome Journal (TM)
Let’s see how the judges were able to easily submit their inputs and then view the recommended decision outcomes.
Each of the judges used their own device (mobile phone, tablet, computer) to provide their inputs within a couple of minutes. When the final submission occurred, Volley ran the hundreds of calculations involved in an AHP analysis, and seconds later displayed the results in both graphical and numeric form.
As mentioned, Volley provides both Decision Science and Behavioral Science capabilities. This is what differentiates the platform from traditional statistical analysis tools that only do the former, or the many team messaging tools that capture comments and sentiments on the latter point. Volley calls this the Outcomes Journal (TM), and it even provides a video interface to log entries about decisions!
The use cases for these Journal entries include conducting debriefs and post-mortems of a decision that has already been completed. For example, Julie’s comments are captured in the Volley system, and then the sentiment is scored and graphed. Journals are also effective for pre-mortems, where the participants share their thoughts about how a decision may turn out… including how they would suggest improving the outcomes of that decision.
This technique makes the process of decision-making a closed-loop process… leading to better outcomes over time, and more aligned teams along the way.
Speaking of judges, it is often appropriate and even necessary to identify that judges (aka. decision-makers) have different levels of expertise in a field. For this Hackathon event, we determined that each of the judges had an equivalent level of expertise, since two were tenured representatives from their respective non-profits, and the third had significant experience doing technology consulting for non-profits. But if this were not the case, and there was a need to weight the judges’ inputs differently, that would be important in the selection of a Decision Science platform. Volley allows for participant weighting via “sliders” similar to the AHP input wizard…
In conclusion, the CU Low-Code Hackathon was a success thanks to the outstanding talent and collaboration of the students, and the dedicated involvement of the non-profit partners. Volley did what it was built to do… the heavy lifting of the statistical analysis and collaborative decision-making. Despite the evaluations being time-sensitive, and the criteria subjective, the panel of judges was able to arrive at a clear and confident decision.
The event was generously sponsored by Skuid, and the continued support from Salesforce.com and the Leeds School of Business have been instrumental in the strong growth of the CRM and Low-Code programs at CU Boulder.
Congratulations to all the teams. Examples of their work are available on YouTube, Vimeo, and by special request. One last example of the sheer creativity on display during the event was the following piece of functional art. QR codes have become more visually appealing lately, but wow, Team 7’s hand-made QR code for A Precious Child is off-the-charts clever… and it actually works!
Volley Solutions is an exclusive partner of V2 (the creator of the HumanCentered.Technology curricula and the Low-Code Hackathon).
#low-code #hackathon #no-code #decisionscience #decisionquality #behavioralscience #decidebetter
