Today I pitched my idea on visualising production data so that the team can see what kind of experience our customers are having. This is a brief update on my progress…
I invited my fellow Tester, my Test Manager, my Dev Lead & the Dev Manager – All of them turned up, what a great start! I kept the demo to 30 minutes as I wanted to keep it brief & to the point. My starting point was that us Testers are spending far too long manually testing in production.
What made me mad
What made me sad
- I’m not convinced I fully got my point across about the benefits.
- We had a conversation about graphs being good for retrospective trending, not for live monitoring, but I mentioned the idea was retrospective trending over the half / hour period of the release but that didn’t wash.
- My 2nd / 3rd order ignorance of our metrics & how they are measured really came to light.
- My ignorance of New Relic & Splunk showed in the demo – even though I talked about what I had planned with condensing the dashboard into fewer graphs & that this was a Proof Of Concept the demo could have been more polished.
- At times I felt a bit like they were thinking “Why is a Tester doing this?” Perhaps it was just my paranoia. As its my role being impacted, I definitely feel it is up to me to try & sort, hopefully with the support of my team.
What made me glad
- I haven’t been told to stop!
- Everyone in the room appeared to like the idea & agreed that there definitely was some scope for this kind of dashboard.
- They questioned why I chose the metrics I did & I received some great feedback about the sort of metrics we should be looking at & how both New Relic & Splunk can be used in conjunction to serve this purpose
- I got some ideas on visualisation other than graphs. This was interesting. I was initially reserved about the idea, but I think a graph might be well complimented by a gauge or a dial (Splunk)
- I got some pointers on the path I should be taking now
- I got great support from my Test Manager for the angle of reducing testing time in Production as a primary driver / motivation for the dashboard.
- Plans are afoot to get Apache data into Splunk (as well as other key logs)
- Talk of looking at other monitoring tools (guess what, I’m pushing Graphite)
Aside from the demo, I enquired about getting another machine to do my experimenting on. This went down well in that I haven’t been denied, but due to a limitation in the number of ports available they’d rather I use a virtual machine. If it does the same job, I’m not fussy. Previous Actions – actions in green have been completed
- Find out why we’re not using New Relic RUM
- Find out if New Relic RUM might actually provide value for us
- Learn how to include several metrics in one graph to save space (e.g. logins, deposits, bets placed)
- Demo my idea / current dashboard to the relevant people
- Take feedback / ideas on how to improve the dashboard
- See if we’re ever going to get Apache data into Splunk (I know the Sys Admins want some other data logging as well)
- If successful, get some monitors up on the walls
- look into setting up the dashboard / monitor on a web server & embed the charts in an HTML page to see if that saves space
- Having the dashboard / monitor on a web server will help with different people / locations being able to see the dashboard via a URL
- Investigate Reload & Revolver to see if they are having memory problems or find an alternative
- New Relic RUM didn’t come in the conversation – I’d like to look into that further
- Investigate some other metrics & how they relate to different levels of the technology stack (without losing focus on customer experience)
- Investigate some other ideas for visualisation of data
- Acquire physical / virtual machine to test my dashboard on
- Getting dashboard on a big screen on a wall is still an action, but now its a bit further off…
Finally, work is ramping up a bit so I need to focus on this side project somewhat less. Unfortunately I will be without family at the end of January as the wife & girls return to the UK to catch up with family & friends. I am going to use that time to create a much slicker dashboard which really sells my idea. Catch up with you on this in February, hopefully with something quite productive…