Mission Hack- Cognitive Services, Bots, IOT and gaming fun

First published on MSDN on Feb 20, 2018

Guest blog by team Mission Hack from Cambridge Hack by Johannes Goslar Microsoft Student Partner at Oxford University

So the idea

In the beginning, we had a motivated team of 4 randomly mixed together people, set out on a mission to not cure cancer, but visually identify different skin conditions, we built a hardware prototype and reduced the cable clutter of a raspberry pi when attaching a camera, speakers and a microphone.

The Pivot

But we only had 24 hours, we needed more data and more time to do that. So then one had 4 slightly less motivated people frantically searching for another idea integrating their newly learned custom vision experience and their refreshed chat bot construction.

The new Plan

After some Red Bull, the usual backup plan looked more and more appealing: Build a game with that. In 18 hours. (While getting some sleep). Some brainstorming (and inspiration from previous games at other hackathons (game designer standard procedure: copy/modify/mutate)) brought us to the final conclusion: “Mission Hack”.

As the game is location bound, you sadly cannot immediately jump into it, but watching this video hopefully helps:

Mission Hack Summarised:

All participants get the same new mission every 5 minutes.

What’s the mission: Take a picture of the asked item, which might be: stress balls, google keyrings, crisps backs, red bull stacks, fire exit signs (safety is important), Microsoft swag, toilet signs…

We trained the custom vision endpoint to differentiate these (and notice if you take a picture of a picture).

The first person to do so gets a point and no one else can claim the current mission.

Wait for the next mission.

Repeat.

Let me show you our fabulous pitch

Our treasurehunty design based on old books (and WordArt).

Mission Hack will grow soon (at the next hackathon) to allow the full takeover of all events you are visiting.

It’s 2 simple steps

Find the items

Locate the beacon

Crack the…

It’s 3 simple steps

Find the items

Locate the beacon

Crack the code

The Code

Which code you may ask now, this code:

No hackathon is complete without some hardware, so instead of finding items and taking the picture, another mission type was to find the hidden Raspberry Pi and decipher its binary code.

Now you are probably asking, how did that all work together. Simple Solution: Microsoft Azure.

Mission Hack is basically a plain chat bot, built on the Function Bot template. The custom vision service is accessed via http calls, and as the function bot itself is a standard function app, it was easy to add another http function the pi could connect to.

The bot framework made it quite easy to connect it to the events slack and our fresh Facebook page. It sadly couldn’t convince the Facebook people at the hackathon to review our bot and enable it for public audiences, we had to do that on our own.

So how does the code work:

Well, at least, this is the most important bit. Bugs are mine, no code review was done, it did the job for the moment, don’t replicate at home. (Or if you want to do so: https://github.com/TernaryMed/ternaryBot (ignore the commit log))

Back to the presentation:

Top right: Jeff, our ML sorcerer from Edinburgh, also handling the mysterious beasts of the function app/slack/Facebook connection wizardry.

Top left: Johannes from Oxford, absolutely not likely to drink some Red Bull as evident in the first picture of this blogpost

Bottom left: Matt, our metal bending and resistoring Raspberry Eater from Cambridge, occasionally turning on LEDs by pure will power (or pressing the on/off-button)

Bottom right: Liz, tamer of the Custom Vision incantations and definitely the person who took the most images of all hackathon visitors

If you were a participant at the hackathon you might still try out Mission Hack at the event’s slack. And if you think, that definitely didn’t work, you are not totally wrong as evident:

Notice to self, don’t add random bots to general channels at 3am without testing or explanation (or at all).

What we Won

ARM liked our use of hardware and we won the ARM hardware prize, which will enable even more interactive hardware next time.

Microsoft liked our use of Azure, awarded use the Azure Challenge prize and made sure we continue using Azure, which truthfully didn’t take any convincing at all. I will definitely use it for the next hackathons.

Hackathons

If you are interested in using Azure, making a game and joining my team (and also are going to START HACK in February or VHACKS in March), feel free to message me at https://twitter.com/ksjogo