It's not always easy for product owners and their team to prioritize technical initiatives over business ones, and in time documentation and dependencies are outdated, code is increasingly difficult to maintain, new joiners have a hard time getting started, knowledge is lost… At HelloFresh, Chapter Days help with all this and more. Maybe they can help you too.

Foreword

Before we get started, we should probably define what's a Chapter and what is Technical debt, as we'll use them throughout this article.

What's a Chapter?

HelloFresh is organized à la Spotify, with Tribes and Squads. Chapters group specialties together. We have backend chapters, frontend chapters, mobile chapters, QA chapters…

I'm the backend chapter lead for the tribes Value Added Services and Weekly Active Customers. Even though the tribes have different missions, they share the same goal of retaining customers. Together, they count 20 backend developers.

The role of chapter lead at HelloFresh is similar to the role of lead programmer in other companies. In a few words, I'm responsible for developing people, planning technical road maps, overseeing technical changes, driving technical excellence, spreading knowledge throughout the organization… More importantly, I'm here to help.

What is Technical debt?

Required changes that are not completed are considered debt that must be paid at some point in the future. Just like financial debt. You'll find an in-depth explanation on Wikipedia or on Martin Fowler's blog, but here are some common examples of technical debt:

  • An experiment has ended but its code has not been cleaned up or removed.
  • A change has been made to the codebase but is not reflected in the documentation.
  • Dependencies are old, some abandoned by their authors.
  • A version of a service/language/standard in use is being retired.
  • The codebase is increasingly difficult to change or extend.
  • More features are added to services instead of creating new services.
  • Shortcuts were taken to implement a feature.

How technical debt accumulated at HelloFresh

At HelloFresh, technical debt accumulated because we didn't keep our dependencies up to date; we launched many experiments but didn't clean up after they concluded; we added too many features in single points of failure; we lost knowledge and couldn't say if a feature was obsolete and could be removed; we introduced micro-services but maintained coupling; we provided new services but parts of the system were not updated to use them; we took shortcuts implementing features to meet delivery timelines… Nothing out of the ordinary really, especially for a tech company that grew so fast in a few years.

Technical debt was hindering our development, tasks were taking longer to complete and were often adding more debt. It was a source of frustration for developers, especially new joiners, who had a hard time figuring out the complexity of the codebase. Technical debt was costing us time, and therefore money.

Genesis of the chapter days

For a while, we tried allocating 10% of the developers' time to pay technical debt, but without proper motivation or well-defined objectives, few ever used that time. To find something more effective we organized a meeting with the chapter leads and the CTO. A colleague suggested using the chapter meetings, but one hour every two weeks was hardly enough for such a challenge. Also, I didn't want to sacrifice the presentations we had—and still have—during these meetings because they are great opportunities to share knowledge. That's when I suggested consolidating and formalizing this 10% of developers' time as Chapter days.

What's a chapter day?

The main goal of chapter days is to pay technical debt. But, as we go, we also include exploratory initiatives and try to keep up with new technologies, standards, and best practices. For instance, we tried a GraphQL implementation of an API, we used contract test between two services, we documented our events using AsyncAPI… Additionally, we have brainstorming sessions to refactor components or extract them as services. Moreover, we have code audits to find areas of improvement in our applications, especially applications inherited from other tribes.

Chapter days occur every other Tuesday. We chose Tuesday because it's one of the deep-work days at HelloFresh, when no meetings are allowed. This is common to all chapters, so we can organize frontend/backend pairings when we need to.

We encourage pair-programming, as a matter of fact, we often pair people that don't usually work together. We believe it helps foster team spirit, spread knowledge, and shake up habits. It also enlarges the focus from the squad to the tribe, for a bigger picture.

Organizing a chapter day

Each chapter organizes according to their needs. The following description is how I organize chapter days with my team.

The week before the chapter day, together with my deputy and a TPM, we review the tasks that are on the chapter's board. We select several and try to come up with objectives, usually in the form of epics.

We create a page in our Wiki and list the objectives and the tasks. The page also features a placeholder for keynotes, which will be collected later, near the end of the chapter day.

We share the page on the chapter's Slack channel for feedback. We might change a couple of things, like team compositions. Then the chapter day is validated and ready to go.

The page of our first chapter day
The page of our first chapter day

A typical chapter day

A typical chapter day has four key moments: a short kick-off presentation to start the day; work-work-work; gathering of keynotes; a short accomplishment presentation to end the day.

We don't have stand-ups on chapter days. Instead, we meet at 10h for a short kick-off presentation to go over the main objectives for the day. Following the presentation, we have a brief Q&A session for any clarification needed and then the teams assemble and start working in a Hackathon fashion.

The teams work all day long on their ticket and when they are done they can ask for another ticket. The chapter lead and the deputy contribute to the effort as well. They also go from team to team to offer assistance and monitor progress.

At 16h30 we start collecting keynotes on the chapter day's page. They include progress reports, accomplishments, struggles, findings… They will be used for the closing presentation and to help keep everyone informed about the work we did that day and help us decide what we should do next, as we often get ideas for improvements.

At 17h30 all chapters and interested colleagues gather for the closing presentation where each chapter lead does a 10 minutes presentation about the accomplishments of the day. We try to focus on what could be beneficial or interesting for the other chapters. For instance, a performance improvement on an endpoint that was bothering frontend or mobile, or the idea of a service that could provide a better API for a screen. The presentation is also an opportunity to applaud each other's accomplishments and warm our hearts.

Following up on chapter days

The day following a chapter day, we send an email detailing the accomplishments made by each chapter to the tribes that participated. If they can find the time, developers continue working on the tasks that are almost complete, and the chapter lead or the deputy follow their progress.

Finally, at the end of every quarter, we send an email detailing key accomplishments to the whole company. It helps keep everyone in the loop and have a moment of well-deserved pride. 🌈

Retrospective

We have a retrospective every other 7 chapter days (~3 months). At the time of this writing, we had two retrospectives, and they have been overwhelmingly positive. We appreciate the mix of teams and the Hackathon spirit. We found it refreshing to work outside of daily activities, and we found a great opportunity to share knowledge about projects and services. We were also happy to finally have time to improve the codebase, fix annoying stuff, solve old issues, update our documentation…

After the first retrospective, we defined clear objectives and expected outcomes for each chapter day. We improved the tasks descriptions and agreed on a definition of "Ready". We agreed on participation policy after we noticed that not everybody was effectively participating.

The team asked for more brainstorming sessions, which were included in subsequent chapter days as RFC workshops, brainstorming sessions to extract features as services, and code audits. The team also asked for more challenges, which were included in subsequent chapter days as exploratory initiatives of little known technologies and best practices.

Overcoming challenges

Chapters days are sadly not all butterflies and unicorns, some challenges can tarnish the experience. Here's a couple of them:

  • During the first chapter days, product owners saw an opportunity to have more meetings. I quickly put an end to it by reminding that meetings were against the deep-work policy. I had to ruffle some feathers, but once they saw the value provided by the chapter days, it was never again an issue.

  • Developers going back to their desk and continuing with their daily tasks is a problem. You have to spend a lot of energy keeping these developers engaged. Luckily it's often a habit that only needs time to change. We are experimenting with the seating arrangement, but it's not popular.

  • We don't have a clear participation policy yet for engineer squad leads, and it's very disruptive, especially when you discover during the kick-off presentation that—without so much as a warning—none of them joined the chapter day, although they had tasks assigned. Reshuffling teams and dropping initiatives at the last minute is disheartening. Have a clear participation policy to set expectations and help with organization and planning.

  • Developers might feel discouraged if they don't see the value in what they're doing. You should describe tasks as well as possible and argument their benefits. Also, try to set clear objectives and outcomes.

Overcoming the greatest challenge: the tasks

Not all tasks are interesting, that's just the way it is. We try to find a good balance between tedious tasks and creative initiatives, but it's not always possible. Sometimes we have to update a bunch of configuration files to move services to a new infrastructure. It's not glorious, but it has to be done. Besides, we all like different things: some like fixing things, some like shiny things, some like thinking about things… There's no pleasing everyone, all the time. Get over it.

Try to assign tedious tasks to new joiners, refactoring or documenting are perfect opportunities to get familiar with a codebase. Also, make sure the same developers don't always get the same type of tasks. Just like in cooking, variety is key. More importantly, ask the team to come up with tasks. Remember, the lead is not the chapter, the whole team is.

Accomplishments

After 16 chapter days, here's a short list of our accomplishments:

  • We drafted 4 RFCs to extract services from our legacy API, and they are being implemented.
  • We refactored several sensitive components.
  • We improved the performance of a few endpoints.
  • We removed obsolete features and cleaned up our APIs.
  • We cleaned up our experiments.
  • We updated our documentation: documented our consumers, publishers, events; fixed broken links; defined a standard for our READMEs and updated all of them.
  • We made sure new joiners can run services locally with minimum effort. Now they often contribute and deploy to production on their first week.
  • We updated our dependencies and now benefit from their improvements.
  • We improved our monitoring and alerting.
  • We extended our test coverage and added acceptance tests to all our services. We no longer have services crashing or breaking contracts after a deployment to live.
  • We have circuit breakers to prevent constantly recurring failures.
  • We experimented with GraphQL, Contract Test, and profiling of live applications.
  • We audited some of our critical services and domains and found many areas of improvement.

That's over 250 tasks completed.

Conclusion

Chapter days started at HelloFresh as an experiment of a tribe to pay its technical debt. After numerous achievements, chapter days soon spread to other tribes in the company. They considerably help us pay our technical debt, but also foster team spirit, spread knowledge, and drive technical excellence.

On a personal note, chapter days are the tool I was missing to efficiently organize technical work, and the means to improve our systems regularly. They also help me grow considerably into the role of chapter lead.

So… When are you starting your chapter days?