After a year of good and loyal service the Redis instance used by my service went down—rebooted by AWS for some unknown reason—making the service inoperative for about 30 minutes. A situation that could have been avoided if the service was resilient to cache unavailability.

The life time of a request

The service implements the command dispatcher pattern, for each request a corresponding query/command object is created, which is then passed to a query/command dispatcher to be executed. The result is serialized and used as the response body. To speed up queries, an event is emitted before their dispatching, if a listener provides a cached response it is returned immediately, short circuiting the dispatch/serialize process. To keep the cache consistent with the database, an event is emitted when a command is dispatched to invalidate cache keys.

It's during the dispatching of these events that the application would crash. Not because calls to the cache would fail—I already had try/catch blocks handling those cases—but simply because the cache was instantiated while the connection to Redis was broken.

The intricacy of dependencies

Event listeners related to cached responses depend on a ResponseCache instance. The class exposes simple methods to retrieve and store responses using their corresponding query/command as key. The dependency graph could be represented as follows:

ResponseCache
└── TagAwareAdapter
    └── Adapter (items)
        └── Redis
    └── Adapter (tags)
        └── Redis

Because I'm using the same Redis instance for both items and tags, I figured I could maybe decorate that instance with try/catch block around it's public methods, but it proved to be impossible because Symphony's adaptor expects a very specific set of classes.

Since I could not decorate Redis, the next best thing was the ResponseCache class.

Making the service resilient to cache failure

In order to avoid instantiating Redis during the initialization of the dependency graph, I created a proxy to ResponseCache, that would only require the actual service when one of its method is invoked. I started by renaming ResponseCache as BasicResponseCache, then I added a ResponseCache interface. Because I don't use the dreadful Interface suffix I did not have to modify my code base, only to update my services configuration.

And that was it!

<?php

namespace HelloFresh\RecipeService\Presentation\HTTP;

use Psr\Container\ContainerInterface;
use Symfony\Component\HttpFoundation\Response;

class ResponseCacheProxy implements ResponseCache
{
    /**
     * @var ContainerInterface
     */
    private $container;

    /**
     * @var string
     */
    private $service;

    /**
     * @var ResponseCache
     */
    private $instance;

    public function __construct(ContainerInterface $container, string $service)
    {
        $this->container = $container;
        $this->service = $service;
    }

    /**
     * @inheritdoc
     */
    public function retrieve(object $query): ?Response
    {
        return $this->getInstance()->retrieve($query);
    }

    /**
     * @inheritdoc
     */
    public function store(object $query, Response $response): void
    {
        $this->getInstance()->store($query, $response);
    }

    /**
     * @inheritdoc
     */
    public function remove(object $command): bool
    {
        return $this->getInstance()->remove($command);
    }

    /**
     * @inheritdoc
     */
    public function clear(): void
    {
        $this->getInstance()->clear();
    }

    private function getInstance(): ResponseCache
    {
        return $this->instance ?: $this->instance = $this->container->get($this->service);
    }
}
  HelloFresh\RecipeService\Presentation\HTTP\SimpleResponseCache:
    arguments:
      $cache: "@cache.response"

  http.response_cache:
    alias: HelloFresh\RecipeService\Presentation\HTTP\SimpleResponseCache
    public: true

  HelloFresh\RecipeService\Presentation\HTTP\ResponseCacheProxy:
    arguments:
      $service: http.response_cache

tl;dr

Consider using proxies for dependencies that could bring your application down during their instantiation.