To say it's been an eventful 4.5 days since the launch of Make Echoes would be a bit of an understatement. The Twitter response from the Twitch streamer base was honestly amazing. Special respect goes out to the Destiny directory which really did a lot of lifting on amplifying the social reach of this launch.
As it stands right now, 20 of the 100 Alpha slots are spoken for and honestly, I haven't done half of what I wanted to for the initial launch. A lot goes into holding some of those actions in reserve but the biggest reason was that I wanted to make sure the service worked as anticipated. This week the service went from some 7 test streamers to 29 total users and it started to show some strain and some incredibly odd behavior under the additional load.
That said, between the test streamers and the new customers 45 new YouTube videos were produced since Wednesday's launch. With less than a half dozen showing some kind of unexpected result and most of those were resolved and re-uploaded within 24 hours.
As I think I've made clear, this is still very much and Alpha and Thursday morning I woke up to just over 200 emails from my server complaining about not being able to upload a video. My YouTube API quota is well below the 200 video uploads a day so I knew that upon opening my inbox and seeing those emails that it was going to be a very rough day.
Ends up that through a completely unexpected series of events, some string manipulation in relation to YouTube posting data created an invalid entry in the YouTube description. The status code and error message that were being sent back by YouTube were ones I had never seen before and therefore the tool didn't have a way to process them. So Make Echoes being the good bot that it is, kept trying to send the stream recap video that it had created up to YouTube. This is how it blew through the entire day's quota in about 45 minutes.
I've put some restrictions in place to prevent this from happening again including a localized version of the YouTube Quota where I actually deduct 1605 points vs. YouTube's 1600 just to make sure it always stops short and a counter that after the third upload attempt it kills the entire script. Additionally, I've made sure to do a final REGEX against the title and description strings in order to eliminate any of the kinds of characters that can produce this result again.
This is exactly why I wanted a limited number of users work with here at the start. Uploading a dozen videos at between midnight and 6A to make up for a lost day's uploads is fine. Doing 100 would have been a stretch. Doing 1,000 likely would have broken me and the way the system is set to create these videos. Hoping that the YouTube Quota is never a problem again though given all the changes.
Shortly after the first users signed up for their accounts and the Chat bot got in their chat rooms, the CPU utilization went from <2.5% to 40% and then held in that position. While I was handling a bunch of issues for customers and then the YouTube Quota Exceeded event, it was in the back of my head that some of the other weird behaviors I was seeing from the server could be related to this high CPU rate.
The first two attempts to nail down the source of it were completely unsuccessful. Friday evening, I made the decision to bring the service down the next morning to try and really sort it out. Fortunately, I went to bed and woke up with a new idea to check and sure enough 18 additional characters in the right file on the server and a quick full reboot instantaneously produced being fully connected to all the chat rooms, doing renders and file movement in the expected CPU usage ranges. Most importantly, when the server wasn't under large user load or rendering a video it was back at less than 3% CPU usage 92%+ of the time.
With the server in a good spot and the first customers creating videos that were successfully posted to their YouTube channels, I spent yesterday afternoon and evening tackling the two biggest feature requests that I had heard from the test streamers and the new customer base: The ability to remove the last clip and the ability to give viewers the ability to add clips.
The !echoundo or !echodel commands are actually incredibly simple in practice but when doing testing on one of my own streams I found that the repeated use of them could create loss of clips that one might actually want to keep. Accordingly, this feature is limited to clips generated in the last 5 minutes. This gives someone the ability to remove a known bad clip in the chat without having to go to the back end of Make Echoes and delete it. By limiting it to just the clips generated in the last 5 minutes, it limits the damage that a single mod/approved user could do should they go rogue.
Since day 2 of using Make Echoes @SheSnaps has asked for the ability to have non-mods add clips to the stream video. I had hoped to sneak this one in before the release but it just didn't happen since I ended up having to build the website three different times (war stories for another day). That said, this is another simple to perform feature if one already has the API frameworks tested and the chat bot pushing and pulling data. Sure enough after about 3.5 hours I was able to get it functioning and tested it last night without too many issues so I decided to throw it into production and allow the user base to use it.
!echoadduser + the target twitch name or !echouseradd + the target twitch name will now allow a normal viewer to be capable of having their suggestions added to the stream recap video.
!echoremoveuser + the target twitch name or !echouserremove + the target twitch name will remove them from being capable.
Additionally, for users who do not have permissions I now have the @MakeEchoes chatbot responding in the chatroom that they don't have permissions. The hope here is that if they know that they can't add that they won't spam the chat room.
The current list of features that I want to have built before I call this a 0.2 was 23 items long. With the work put in since the launch, that list is now down to 16. Here's what I've got my eyes on for this next week or so:
Overall, with where I was 8 days ago and feeling uncertain of both the tech and whether or not it had a potential user base to where this all is now, it's a hell of a turn around. And the energy that the user base and those looking on via social media have provided has only made what I lovingly call an "obsession project" (not a passion project since this has literally taken over my conscious hours in a way nothing else ever has) take over even more hours and having me miss out on more sleep to get all the pieces right. It'll calm down soon as I go back to working on contract work but for now I'm rolling with it.
Go and be awesome in the world. We all deserve it.
