Now that we can detect a game pretty reliably and fast, the next step is to see if we can apply this technique to a live stream. There are a few things that need to happen in order to get video stream, get the stream, extract the frame and predict. Since video is just a series of images, we can run a prediction on each image and possibly get a more accurate over all result.
The whole lifecycle from video to prediction:
- Get stream playlist from Twitch
- Download each video from playlist
- Concatenate into a single video clip
- Extract one screen per 60 frames (every 2s or so depending on the stream)
- Extract the histogram of the screen
- Run the historgram through our prediction Model
- Count up what game was predicted the most and we have our result
There is one additional gem needed to accomplish this and its the ruby-ffmpeg. This is only available via the Github repo (it’s not in rubygems) and you’ll need ffmpeg installed.
To try this on any stream, just pass in a TwitchTV url in the command line. This is a CS:GO stream.
$ ruby stream.rb http://www.twitch.tv/fattypillow Downloading http://video26.lax01.hls.ttvnw.net/hls62/fattypillow_19837787584_407982423/high/index-0000003017-Z85m.ts Downloading http://video26.lax01.hls.ttvnw.net/hls62/fattypillow_19837787584_407982423/high/index-0000003018-VURZ.ts Downloading http://video26.lax01.hls.ttvnw.net/hls62/fattypillow_19837787584_407982423/high/index-0000003019-WDBZ.ts Downloading http://video26.lax01.hls.ttvnw.net/hls62/fattypillow_19837787584_407982423/high/index-0000003020-GApy.ts Downloading http://video26.lax01.hls.ttvnw.net/hls62/fattypillow_19837787584_407982423/high/index-0000003021-52wI.ts Downloading http://video26.lax01.hls.ttvnw.net/hls62/fattypillow_19837787584_407982423/high/index-0000003022-IiTb.ts Predicting frame @ 536037.03 Predicting frame @ 536127.03 Predicting frame @ 536216.985 Predicting frame @ 536306.985 Predicting frame @ 536397.03 Predicting frame @ 536487.03 Predicting frame @ 536576.265 Predicting frame @ 536667.03 Predicting frame @ 536756.265 Predicting frame @ 536846.985 Predicting frame @ 536936.265 Detected: csgo Prediction results: csgo: 8 minecraft: 2 hearthstone: 1
For a Hearthstone stream:
$ ruby stream.rb http://www.twitch.tv/itshafu Downloading http://video26.lax01.hls.ttvnw.net/hls-6ecba8/itshafu_19841113280_408079164/high/index-0000001238-Yzxx.ts Downloading http://video26.lax01.hls.ttvnw.net/hls-6ecba8/itshafu_19841113280_408079164/high/index-0000001239-wQjg.ts Downloading http://video26.lax01.hls.ttvnw.net/hls-6ecba8/itshafu_19841113280_408079164/high/index-0000001240-LCeM.ts Downloading http://video26.lax01.hls.ttvnw.net/hls-6ecba8/itshafu_19841113280_408079164/high/index-0000001241-VMUc.ts Downloading http://video26.lax01.hls.ttvnw.net/hls-6ecba8/itshafu_19841113280_408079164/high/index-0000001242-v3ns.ts Downloading http://video26.lax01.hls.ttvnw.net/hls-6ecba8/itshafu_19841113280_408079164/high/index-0000001243-udLl.ts Predicting frame @ 114013.44 Predicting frame @ 114103.44 Predicting frame @ 114193.44 Predicting frame @ 114283.44 Predicting frame @ 114373.44 Predicting frame @ 114463.44 Detected: hearthstone Prediction results: hearthstone: 6
Of course just like with the previous part 1 and part 2, results can vary. Especially if the streamer’s feed isn’t on the game or is too distorted with other stuff on the screen. The advantage of doing this with a video stream is that we can have many more predictions so having a couple false positives in the mix wont throw off the entire result.
The entire codebase with the dataset for this experiment can be found in the Github repo.