Kickass, Inc. Tech Pulse

Mar 29th, 2015 Weekly CTO Report

Cached Mar 31st, 2015

This week brought one unexpected challenges. As expected from the prior week's observation, our new Node developer, John Smith, had a difficult time assessing his workload this week, and thus was over commited. As a result, a buggy patch was sent to QA last week, and our QA team failed to catch a database validation error. This patch was pushed to roduction Saturday morning, which resulted in over 1000 database write errors. PagerDuty successfully notified me of the errors, and I reverted to the old patch within 20 minutes of the event. I will be making a report of suggestions for our new developer, and our QA tester, along with writing tests against the bugs that slipped through. Our other new developer, Mark Jones successfully completed his weekly sprint commitments, although he was up late Saturday night to do so.
Despite the errors on Saturday, our stack had a 99.997% overall uptime. Our app suffered an outage Wednesday afternoon at 12:45:46 GMT, due to a Level3 failure at the Rackspace data center. The HAProxy load balancer dropped for 32 seconds Friday night at 21:57:38 GMT, so I will be adding a measurement of the uptime of that server to this report next week. All feature goals and bug fixes in this week's sprint were completed on time, and are in QA this week. I do not anticipate any issues, so the video profile upload feature many customers are asking for, should launch on time next Friday.

Dev	Ontime Bug Fixes	Δ Last Week	Ontime Features	Δ Last Week
Sally Rider	100%	+2%	100%	+4%
John Smith	72%	-19%	45%	-34%
Mark Jones	100%	+5%	100%	0%
Neil Armstrong	94%	-5%	92%	+1%

Client Response Time

VisuHire Uptime

VisuHire Error Rates

89% - On Time

Video Profile Milestone (3 sprints total)

56% - On Time

LinkedIn Integration (4 sprints total)

24% - 1 Sprint Behind

3 Way Conference Calling (6 sprints total)

Weekly Commits

This Week 34
Last Week 28

Javascript

12,567 lines
Up 567 Lines

Bug Fixes

This Week 22
Last Week 18

Python

13,354 lines
Up 1,267 Lines

Open Bugs

This Week 14
Last Week 17

Roach Events

This Week 1
Last Week 0

P0: Identify source of QA missing database write bug. Document, and make appropriate changes to QA procedure to prevent future missed bugs.

P0: Collect list of all affected users from Staurday's outage, draft message, and send with apology, explanation, and steps that we've taken to prevent future outages.

P1: Skype feedback meeting with John Smith to develop action items for him to better plan his workload, and avoid overcommiting in the future.

P1: Add HAProxy load balancer to NewRelic monitoring, and generate uptime metrics for future CTO reports. Identify source of downtime, change stack and fireplan accordingly.

P2: Finalize proposed changes to tech stack for Feb 12th board meeting, along with monthly budget, and outline reasons for the new redundancies.