How our own app broke at the H-hour of our main event
Probably the most cherished thing we have at the company is our own hackathon platform, Junction App, where we use the Gavel algorithm to judge projects. And yet, despite all the love and care our tech team gives it each day, it somehow let us down at the most crucial moment of this year’s main event.
Three weeks ago, we were at the final stretch of the Junction hackathon weekend. Our ears were blessed and nerves were calmed with a developer-friendly meditation track, whispering comforting hackathon-related phrases through our headphones. “Gavel review, works”
Yeah, until it doesn’t.
A recap of the storm
A virtual event with thousands of people from all around the world is likely to encounter a glitch or two, but the timing for this one was probably the worst possible.
The submission deadline was at 9 AM (Finnish time) and the peer reviewing was set to begin 15 minutes later. But when the participants entered the reviewing, they encountered a message saying they were done with the task, even though a not single project had been reviewed or judged.
After troubleshooting for a while, the bug was found. The issue was caused by a team deleting and recreating their team during event, which left their previous project dangling with a null reference, which was enough for the Gavel reviewing system to give up even before starting.
50 mins later, the Junction App was back up and people could start the reviewing process.
So why do it like that and not just use judges? Wouldn’t it be easier?
Well, yes and no.
We take pride in our ability to provide a fair judging process for our hackathon participants. The pairwise-comparison-based Gavel algorithm, where the participants review each other’s projects, is the best one we have had so far. The system incorporates a lot of research on mathematical psychology along with fancy math, resulting in a fair and scalable reviewing system.
It is something a traditional judging system with a jury and a pre-set scoring scale can’t do.
To put Gavel in short: the user is always given two projects to compare, asked to vote for the better one and compare the newer project to the next one (and so forth). This way, one doesn’t have to take in consideration what the absolute perfect, nor what the absolutely worst project would look like. It’s enough to decide the better one of the two that the system suggests.
In addition, participants are naturally curious to understand why a certain project has won. Peer reviewing gives them the possibility to have an overlook of other projects and cast their vote for the winners.
If interested, you can read more about the benefits of pair comparison in reviewing here and about it being implemented at Junction events here and here.
Lesson learned: you can never test something enough
We actually did test the reviewing system before using it for Junction 2020 Connected (as we always do before any event). We just did not test it hard enough. The thing is, here the “we” means our full-time team of 9 people. Junction 2020 Connected had 2000 participants.
Yes, we will improve the testing in the future.
P.S. Junction App is an open source platform, so in case you want to help us prevent hackathon hackers going like this (╯°□°)╯︵ ┻━┻, you can contribute on GitHub.
