Visualising Production Data For The Team – Retrospective 1

Today I pitched my idea on visualising production data so that the team can see what kind of experience our customers are having. This is a brief update on my progress…

I invited my fellow Tester, my Test Manager, my Dev Lead & the Dev Manager – All of them turned up, what a great start! I kept the demo to 30 minutes as I wanted to keep it brief & to the point. My starting point was that us Testers are spending far too long manually testing in production.

What made me mad


What made me sad

  • I’m not convinced I fully got my point across about the benefits.
  • We had a conversation about graphs being good for retrospective trending, not for live monitoring, but I mentioned the idea was retrospective trending over the half / hour period of the release but that didn’t wash.
  • My 2nd / 3rd order ignorance of our metrics & how they are measured really came to light.
  • My ignorance of New Relic & Splunk showed in the demo – even though I talked about what I had planned with condensing the dashboard into fewer graphs & that this was a Proof Of Concept the demo could have been more polished.
  • At times I felt a bit like they were thinking “Why is a Tester doing this?” Perhaps it was just my paranoia. As its my role being impacted, I definitely feel it is up to me to try & sort, hopefully with the support of my team.

What made me glad

  • I haven’t been told to stop!
  • Everyone in the room appeared to like the idea & agreed that there definitely was some scope for this kind of dashboard.
  • They questioned why I chose the metrics I did & I received some great feedback about the sort of metrics we should be looking at & how both New Relic & Splunk can be used in conjunction to serve this purpose
  • I got some ideas on visualisation other than graphs. This was interesting. I was initially reserved about the idea, but I think a graph might be well complimented by a gauge or a dial (Splunk)
  • I got some pointers on the path I should be taking now
  • I got great support from my Test Manager for the angle of reducing testing time in Production as a primary driver / motivation for the dashboard.
  • Plans are afoot to get Apache data into Splunk (as well as other key logs)
  • Talk of looking at other monitoring tools (guess what, I’m pushing Graphite)

What else?

Aside from the demo, I enquired about getting another machine to do my experimenting on. This went down well in that I haven’t been denied, but due to a limitation in the number of ports available they’d rather I use a virtual machine. If it does the same job, I’m not fussy. Previous Actions – actions in green have been completed

  • Find out why we’re not using New Relic RUM
  • Find out if New Relic RUM might actually provide value for us
  • Learn how to include several metrics in one graph to save space (e.g. logins, deposits, bets placed)
  • Demo my idea / current dashboard to the relevant people
  • Take feedback / ideas on how to improve the dashboard
  • See if we’re ever going to get Apache data into Splunk (I know the Sys Admins want some other data logging as well)
  • If successful, get some monitors up on the walls
  • look into setting up the dashboard / monitor on a web server & embed the charts in an HTML page to see if that saves space
  • Having the dashboard / monitor on a web server will help with different people / locations being able to see the dashboard via a URL
  • Investigate Reload & Revolver to see if they are having memory problems or find an alternative

Whats Next

  • New Relic RUM didn’t come in the conversation – I’d like to look into that further
  • Investigate some other metrics & how they relate to different levels of the technology stack (without losing focus on customer experience)
  • Investigate some other ideas for visualisation of data
  • Acquire physical / virtual machine to test my dashboard on
  • Getting dashboard on a big screen on a wall is still an action, but now its a bit further off…

Finally, work is ramping up a bit so I need to focus on this side project somewhat less. Unfortunately I will be without family at the end of January as the wife & girls return to the UK to catch up with family & friends. I am going to use that time to create a much slicker dashboard which really sells my idea. Catch up with you on this in February, hopefully with something quite productive…