Posted by: Jeff Timmins | June 30, 2017

Foundations of Release Management

Goals, Success & Details that every Release process should include

Purpose

The purpose of the following information is to provide the reader with a better understanding of the motivations and thoughts behind my general Release Management process. With different cultures and histories behind the Development groups around the world, a process that works for one team may not work for another (i.e. one size doesn’t fit all).

In addition I firmly believe that the Release Management process should not be forced on a Development group but should be tailored or fitted to enhance the model that the team provides.

Because of these beliefs I felt it important to share what I call the Foundations of Release Management, a common fabric that to be used for each set of custom designed Release Processes.      

Goals for Release Management

  1. Help Development groups get their changes to Production efficiently and successfully
  2. Ensure controlled, systematic and documented Software Releases
  3. Use readiness checklists based on agreed to standards instead of emotion based Go/No-Go meetings
  4. Communicate details related to Releases so everyone who needs to know is informed
  5. As needed function as a liaison between Development Groups and Operations, Governance, Help Desk & others

What is Success for Release Management

  1. Work with all individuals gracefully and in a respectful manner because everyone is an Internal Customer
  2. Production deployments meet compliance requirements set by Governance
  3. Releases deployed to Production in a timely manner without the need for rollback or causing customer impact (i.e. Production deployment should Do No Harm or Production environment should be as stable or more stable than before the change)
  4. Communicate the right amount of Release related information to the right people at the right time so they can be successful in their roles (i.e. No Surprises)
  5. Limited or no Post-Release issues found in Production

The common fabric of all Release Management Processes

  1. Introductory planning with Development groups (one-time tasks)
    1. Team make-up
      1. Waterfall, Kanban or Agile
      2. Desired Production Release schedule
      3. Review Tools to be used by Development groups, standards for change tickets, source control and Code repository for builds
      4. What to expect for versioning of packages & Releases
    2. Environments
      1. Confirm which environments to be used for Non-Production, Production, Training & DR
      2. Confirm management of deployment schedules for last Non-Production environment (considered Staging), Production, DR and Training
    3. Validations
      1. Confirm validation teams for each Environment, including agreement on usage of non-Development groups
      2. If possible, establish standard validation times and validation processes for each Environment
      3. Establish Release Gates for critical environments (typically covering pre-Stage, Staging & Production), see below for more details
    4. Communication
      1. Main Development group member(s) that Release Management should communicate with
      2. Establish Communication Group that works for particular Application. This could include any combination of Business Owners, Stakeholders, Development group, Help Desk or Internal User group.
      3. Establish information portal to be used for Release Management information
    5. Monitoring & Reporting tools
      1. Understand what is currently Monitored & what type of monitoring available (typically made available via Operations Group, used to confirm Release success & acceptable Deployment windows)
      2. Understand available Alerting (used to confirm Release success) 
    6. Document above agreements
  2. Desired Development group activities participation (in order to collect needed Release information)
    1. Scrum ceremonies as makes sense (as a Chicken)
    2. Scrum of Scrum meetings
    3. WaterFall planning and other update/planning meetings
  3. Release related Communication
    1. Weekly update of upcoming/future deployments to Application Communication Group
    2. Help with communication related to unplanned events or issues
    3. Confirm deployment impact notification is communicated to customers
    4. For deployments – specific notifications related for Application Communication Group
      1. Deployment coming soon
      2. Deployment started
      3. (as applies) Issue with Deployment
      4. Deployment complete
    5. Post-Deployment report providing details of success or failover of the deployment
  4. Release Planning
    1. Regularly scheduled meeting with Development group, providing opportunity:
      1. For Development group to request future deployments not yet known by RM
      2. To review to-do items for already scheduled deployments
      3. To review lessons learned from previous Releases
    2. Documentation of Release details
      1. Key dates for each Release added to a shared Calendar
      2. Add details for each Release to Release information portal
    3. Production Deployment window planning
      1. Analyzing customer impact for each deployment
      2. Creating estimated deploy window timing
      3. Identify which team members need to be involved in Production deployment (see Appendix below for examples)
    4. Leading Development group & Ops deployment planning meetings
    5. Managing Change tickets for environments that RM manages schedule
      1. Create Change ticket 
      2. Requesting needed approval(s) outside of Governance
      3. Requesting approval for Change ticket in a timely manner (i.e. within Governance requirements)
      4. Representing Change ticket at weekly Governance (i.e. CAB or Change Advisory Board) meeting
  5. Managing Release Gates
    1. The output of the Release Gates is a checklist to be used by the Development groups to identify the status or completion of a stage of the Release (i.e. Release is ready to proceed to next environment)
    2. RM tasks during testing in environment before Staging
      1. Identify Release ticket for Release
      2. Confirm Staging in and out schedules as well as Feature Freeze & Code Freeze schedules
      3. Confirm Directions for deployment are available and reviewed by Development group & Operations and request updates as needed
    3. Development group tasks to complete before Release packages deployed to Staging
      1. Confirm Directions for deployment are finalized
      2. Confirm list of change tickets for the Release are linked/associated to the Release ticket
      3. All change tickets and their related sub tasks are compliant and closed
      4. All agreed to Code Testing has passed to agreed to standards (e.g. X% passed, no Sev2/Pri2 or higher bugs introduced)
      5. All agreed to Performance & Security Testing has passed to agreed to standards
      6. If applies, packages are moved to shared Code repository for deployment needs & future reference   
      7. Verify/Confirm contents of build – verify or confirm the code build/package includes all changes expected in the Release (i.e. all code commits from tickets linked to Release ticket are included in the build)
    4. RM tasks before entering Staging environment
      1. Creates Non-Production Change ticket
      2. Creates Draft version of Production Change ticket
    5. Development group tasks to complete before Release packages deployed to Production
      1. List of Known Issues of the Release provided
      2. Rollback procedure has been validated (can be done in Staging or before)
      3. All agreed to Code Testing has passed to agreed to standards (e.g. X% passed, no Sev2/Pri2 or higher bugs introduced)
      4. All agreed to Performance & Security Testing has passed to agreed to standards
    6. RM tasks before entering Production environment
      1. Confirm Code repository is locked down to eliminate additional modification to Release packages
      2. Confirms Production Change ticket is approved
    7. Confirming health and stability Post-Release
      1. Monitor Post-Release Incidents created, identify which are caused by my recent change
      2. Be aware of site impacting issues related to the Solution, typically by having visibility to health checks and/or system monitors & alerts.  
    8. Continually reviewing the Release Gates with Quality teams and Environment team on for possible improvements
  6. Managing Production Deployment window
    1. Create meeting invite before deployment
    2. Lead the deployment call, confirm starting & stopping times of various tasks and status during the deployment window
    3. Confirms Change is completed and dismisses the team
    4. As needed, schedule Post-deployment meeting & documentation
      1. Retrospective of deployment
      2. Update post-deployment details document
      3. Update information portal

Appendix

  1. List of possible groups that would be part of the Production deployment window
    1. Development group members
    2. Operation team members
    3. Validation team members
    4. Support team members
    5. Product Owners, Business Owners and any other group of people that need to be invited (note that providing optional invites is helpful for some people for extra awareness of the deployment)
Posted by: Jeff Timmins | February 28, 2017

Release Management feels like UPS sometimes

When I think about Release Management I try to ways to communicate to others at work so they better understand why I do what I do. Sometimes “Safe & Successfully Production deployments” just doesn’t do it. Then I started thinking … yeah, I’m like a UPS driver!

ups-tracking_35 I makes total sense if you think about it, right? The way I think about it, if I’m a driver I shouldn’t be thinking or caring that much what is in a package, I should just care that it going to the right place, in the time promised … and that the package will not kill me! Really, I mean if the package is radioactive then I’d say the UPS driver would care about it being in his truck (I’m guessing they have Mr. Yuck stickers but I’m not sure). Same thing with some phone batteries and other live animals. Anyway, I see safety like a code change passing test in Pre-Production – if it passes testing before being deployed to Production it is safe.

And about the address, can you image if dispatch said “Just ask Joe where to delivery this package, he knows and then Sally about the big brown one” and then the driver runs around looking for Joe and Sally to see if they remember. This feels like directions to me – you cannot say “deploy the fix, Fred knows it” and hope to have an environment that allows people to go on vacation.

Then when I think about repeat customers all stories I’ve heard about stores doing nice things for the UPS drivers that take care of them I think of how a Release Manager cares for (i.e. protects) the Operations team.

Like all analogies this one breaks down at a certain point but I think get the idea.

 

Posted by: Jeff Timmins | December 2, 2016

Join my Release Management team

dfslogos-dark-blue-3

Looking for a job, experienced or interested in Release Management? I’m building a team to spread Release Management process across Donnelley Financial Solutions.

Currently my team of off-shore Release Managers support 4 different Solutions but we are expanding to support about 40 Solutions! For that effort I have 3 to 4 openings for Releases Managers in multiple US locations. Those locations are

 

Here is the Job Description
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Be part of the Release Management team as it grows to become the central part of managing Releases for Donnelley Financial Solutions (DFS). Responsible for successful deployments, documentation, risk mitigation & communication for Releases as well as Incident Management for the same solution. Responsibilities may cover one major solution, a group of related solutions, or helping our global team with multiple solutions. Most Releases are code related but some also include infrastructure changes.

Responsibilities 

  • Participate in reviews of current Release Process to find and implement improvement as needed
  • Build & foster relationships with Development, QA, Product Management, Business Owners, Product Support and Executives
  • Work with the Development System of Record (typically TFS) and Change Management ticketing system (ServiceNow) to plan, document, communicate Release details as well as prove Release readiness. In addition, provide transparency and visibility throughout the release process.
  • Track & Manage Software releases through Pre-Production environments to Production
  • Manage risks and resolve issues that affect release scope, schedule and quality
  • Conduct release readiness reviews, milestone reviews and business go/no-go meetings while at the same time understanding system and application dependencies
  • Lead work-hours and/or off-hours Production deployments
  • Troubleshoot deployment related issues as needed
  • Perform other related duties and participate in special projects as assigned

Required Skills

The duties and responsibilities described above are the essential functions of the job.  The qualifications below are representative of the knowledge, skills, and/or abilities required.  Reasonable accommodations may be made to enable individuals with disabilities to perform the essential functions.

  • Bachelor degree and with 2-5 years of relevant work experience OR demonstrated ability to meet the job requirements through a comparable number of years of applicable work experience.
  • Experience with System of Record ticketing products, preferring Microsoft TFS & ServiceNow.
  • Experience working with Development and QA teams.
  • Experience working within Waterfall, Lean and/or Agile SDLC processes.
  • Experience working with Build process is preferred.
  • Able to apply broad work experience and knowledge to the Release process.
  • Excellent communication skills with ability to state messages in a clear manner by using language that is easy for others to understand. Able to explain programs policies and procedures in language that is understood by others.
  • Able to apply excellent functional computer knowledge in utilizing Microsoft Windows, MAC, or other technical tools in completing assignments. Able to apply expertise in all the tools or applications used to complete work assignments.
  • Able to modify communication style both formal and informal to match the appropriate level of the audience targeted. Requires strong understanding of the impact of a message on the organization or customer. Able to write with the clarity and precision necessary for the work being performed.

 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Posted by: Jeff Timmins | November 30, 2016

What to call that Release?

A Release is a Release is a Release? G. Stein (revised)

Are names of Releases important? Should the Software/Service Industry care what we call it Internally or Externally?

From my memory of the shrink-wrap days of Microsoft, they put out regular Releases, Service Packs and Hotfixes (KBs). The today’s Cloud World we see fun names for Releases “Winter ’17” or “Serengeti” but Service Packs and Hotfixes are not well published. When you see a version like “10.2.5.101” you get the hint that this is not the original Release but it doesn’t scream “I’m a Hotfix”. Customers of SaaS vendors that throw around Hotfix too many time typically move on to another solution.

Why should we are about Release names anyway? Why can’t we just call each Release brucesBruce? I mean we could call everything Bruce and while that would be fun and easy to remember it might be confusing for normal people. But I digress.

Release names are important! They help us understand and easily discuss what change happened + it sometimes helps understand why the Release was important. Great but what about “fixes” to the main Release – should they have distinguishing names as well? My vote is “Yes please!” for the same reason that Release name or number should be unique – I’d like to know what Major Release that fix was associated with. This comes in handy during Support calls (what version are you using?) or building a Non-Production environment (I need version X in Staging please) or discussing past or upcoming Releases with Staff (Release Y for Project Llama on Saturday, Release 1 for Project Sam on Tuesday).

Great, names are important but what about the term “Hotfix”, is that necessary to use anymore? I have to admit I have used it many times over the years as an Internal label but I’m being challenged to stop that practice. Why was it important to use before? Because I wanted to track problematic Releases and look for trends that caused it. (Due to tracking I was encouraged to decrease my Release schedule in late December and early January, when I did Release quality increased!) That said, my tracking needs don’t require the label “Hotfix” to be shared with 400 Internal Customers. When it is shared, those Internal Customers could start to develop a poor attitude towards the creators of the code. As long as I have a build number or use a label like “update 1” I can track the Releases and keep track of which “update” was planned and which one requires a Root Cause Analysis (RCA).

 

Going forward I’m dropping the Hotfix label when communicating Release names – how about you?

Posted by: Jeff Timmins | June 30, 2016

Release Management because no one is perfect

To really show imperfect I am I’m going to finish this blog Friday

Posted by: Jeff Timmins | May 30, 2016

Invisible Release Management

Can you see your Release Manager? Is “No News” the same as “Good News”?

The other day I was part of a meeting where Managers gave “shout-outs” for a job well done. Maybe I’m a little too self-confident but I kept waiting for someone to mention my name.

Not surprisingly that comment never came. Then I remembered … I’m the invisible man … no wait, I’m the Invisible Release Manager!

I use the term “invisible” because the Release Manager role is not even noticed if the job is done right. Because of that I really shouldn’t think that anyone would notice. In my first Blog entry titled RM Charter #1 I wrote about the same concept but honestly it has been awhile since I thought about it:


“In addition, RM done well is an invisible process to the customer and to the company as a whole.  This goal is definitely applicable to SaaS (customer focused) and thus easily transferable to a SaaS company.”


I love doing my job well and I have to admit I would love it someone actually too the time to notice and provide positive feedback in a group of my peers but I shouldn’t expect it. Hopefully your management or other leaders are giving you positive reviews of your work – if not I encourage you to find a group of peers that will.

 

Posted by: Jeff Timmins | April 30, 2016

Life can be humbling

Life can be a humbling experience. All around us are examples of things we don’t know as well as we would like. This is a reminder that Release Management is no exception to that rule. The key is to understand this is part of the process of adding value to your organization.

I purchased a car needing more work than originally thought (an example of a humbling CVJointconversation with my wife … create idea for another blog … maybe) so I’ve asked my neighbor for help with some of the repairs. I need lots of help as I have troubles changing my own oil! Per a Auto repair shop’s report I tell my neighbor the CV Boot needs to be replaced (example pictured here) and he starts going on about re-manufactured this, consider the source of the re-manufacturing, where to look for it here and my head starts spinning. After two more conversations and multiple internet searches I am finally able to have a conversation about how to make the correct purchase. Cannot wait for the learning experiences related to the actual repair!

Being a Release Manager can be very humbling as well. As an RM I believe it is important to understand at least the basics of the Solution you support. If you don’t then your value to the organization is diminished (important difference between Release Management and Release Coordination – RMs provide more value). Each time I start a new job I have to learn a new the Architecture and process flow. The task of learning can take a few months and my only advice is to be patient and take great notes – in time you will be comfortable.

Lately I’ve been adding more Solutions to my Release Management responsibilities while hiring other Release Managers to back-fill the Solutions I’ve mastered. Because of that I find myself learning a new set of Solutions (i.e. being humbled again). I find it especially challenging to be an expert in one part of the Organization and someone learning from scratch in another part of the Organization. In time I’ll add the value I want but for now I have to be patient and humble.

 

Posted by: Jeff Timmins | February 26, 2016

Why Release Management is important

I heard a story the other day about a SaaS provider (disclaimer – I have no connection to this company) that I thought should be shared. This is a great example of why Release Management is an important part of an SDLC and why I think my job is important.

Here is the story [with comments], names removed to hide the guilty.


The users at “Company A” use a solution from “SaaS Provider B”. The users at Company A reported that they were unaware that SaaS Provider B performed a software update [Strike #1 and Rule #1 – inform thy users!]. Sadly SaaS Provider B also introduced a new bug into Production during the software update that Company A didn’t know about [Strike #2 – not good!].

The impact of the newly introduced bug was that the program didn’t actually save all the important/critical changes when users selected the save function [Strike #3 – I’d call that a Sev 1 Bug]. On top of that, Company A then used the SaaS solution to submit their very important paperwork, thinking their important/critical updates were actually included [yikes!].
In the end Company A had to re-submit their paperwork [embarrassing, costly, etc.]. In addition they decided to look for another SaaS provider. 

Regardless if this is true or not (I’m fairly sure it is), it is a good reminder of why it is a good idea to employ a Release Manager on your team.
Posted by: Jeff Timmins | February 19, 2016

DevOps communication done well

Recently I saw a great example inside my company of DevOps communication done well. It was related to a minor update to our On-Prem instance of TFS 2015. This time I’m one of the customers and it was very nice to see that someone took the time to share “inside Release information” with me & the rest of the team.

This was is what the email communication looked like regarding the schedule:


Having not heard any concerns from anyone about the time window for this upgrade, we will be proceeding to update TFS from: Friday, March Xth 23:00 ET to Saturday, March Xth 07:00 ET.  Scrum Masters, please let your teams know that TFS will be completely unavailable during this time period.


This was is what the email communication looked like regarding the changes:


Update includes new features in Backlog

TFSUpdateExample

  • Improvements on the web portal: commit details summary is easier to read, and an improved experience for cloning Git repositories.
  • Backlogs: multi-select on all backlogs, drag any item to an iteration from anywhere, Add panel to the iteration backlog, line on the burndown indicates actual capacity, configure settings directly, add/remove users in the sprint plan, and multiple activities per team member in planning capacity for a sprint.
  • Kanban boards: query on columns, card coloring, tag coloring, inline renaming of columns and swimlanes, reorder cards when changing columns, configure settings directly, and hide empty fields on cards.
  • Work items and tasks: tasks as checklist, link to branches and pull requests in work items, task board card coloring, and limit values shown for Work Item type in queries.
  • Build: improved access control for resources, improved source control integration, usability fixes in Build Explorer, and parity with XAML builds for label sources and client-side workspace mappings.
  • Testing: export test outcome for manual tests and test result retention policy improvements.
  • Dashboards: 100% customizable with new widgets and multiple dashboards.
  • Version control: use Git and Team Foundation Version Control in the same project, history and getting started
  • Extending: Service Hooks tab available on-premises

More  detailed  information can be found in Release Notes : https://www.visualstudio.com/news/tfs2015-update1-vs Update is installed to non-production environment so you can try it here: http://rrwin-tfsdrpt.eolcorporate.com/tfs/


 

Posted by: Jeff Timmins | February 9, 2016

Loving the Salesforce Trust

Something I’ve always loved about Salesforce is their Trust site. They have made it SF_Calendaravailable for years and it has been my favorite examples of SaaS customer communication done well. They basically say to their customers “we have nothing to hide.”

When I looked at it again the other day I found two features that were new to me. One is the Calendar section and the other is Performance section. Check it out and try to do it for your customer facing solution!

Older Posts »

Categories