Add GitLab Events Platform blueprint

2c0441aa · Grzegorz Bizon · a253d0ba · 2c0441aa
--- a/doc/architecture/blueprints/gitlab_events_platform/index.md
+++ b/doc/architecture/blueprints/gitlab_events_platform/index.md
+---
+status: proposed
+creation-date: "2023-03-06"
+authors: [ "@grzesiek" ]
+coach: "@grzesiek"
+approvers: [ "@", "@" ]
+owning-stage: "~devops::?"
+---
+
+# GitLab Events Platform
+
+## Summary
+
+GitLab's codebase has grown a lot since the [first commit](https://gitlab.com/gitlab-org/gitlab/-/commit/93efff945215)
+made in 2011. More than a decade of evolution made it possible to deliver a
+solution used by millions of people across the world, and also introduced a lot
+of complexity.
+
+One of the best ways to reduce complexity and cognitive load, required to
+understand something, is to build [great abstractions](https://en.wikipedia.org/wiki/Abstraction_(computer_science)
+that are going to make complexity manageable for engineers.
+
+We propose to build two additional layers of abstraction into our monolithic
+GitLab's codebase:
+
+First one to emit any event with a single line of code:
+
+```ruby
+emit Gitlab::Event.new('events/package/publish', package)
+```
+
+Second one to route events within the codebase and out of GitLab Rails by
+simply implementing an abstract gateways that will decide what is going to
+happen with a particular group of hierarchical events.
+
+This will allow us to reduce the complexity of existing events-based features:
+
+1. [Webhooks](https://docs.gitlab.com/ee/user/project/integrations/webhook_events.html)
+1. [Audit Events](https://docs.gitlab.com/ee/administration/audit_events.html)
+1. [GitLab CI Events](https://about.gitlab.com/blog/2022/08/03/gitlab-ci-event-workflows/)
+1. [Package Events](https://gitlab.com/groups/gitlab-org/-/epics/9677)
+1. [GraphQL Events](https://gitlab.com/gitlab-org/gitlab/-/blob/dabf4783f5d758f69d947f5ff2391b4b1fb5f18a/app/graphql/graphql_triggers.rb)
+
+All of these features are implemented separately, even though we could build
+them on top of a common abstraction. Doing that will also allow us to provide a
+solid foundation for building new features, that will increase our efficiency
+and ability to deliver them.
+
+## Goals
+
+Build new abstractions around GitLab events. This might allow us to:
+
+1. Improve engineering productivity by abstracting away complexity and reducing
+   cognitive load.
+1. Build new, solid foundation aimed for extending GitLab with new features,
+   like GitLab CI Events.
+1. Refactor existing features to use new abstractions, in order to multiply the
+   value provided by existing instrumentation around events (Audit Events,
+   Webhooks).
+
+## Challenges
+
+## Proposal
+
+### Build a common way to emit events
+
+Making it very easy for engineers to extend GitLab codebase with additional
+events being emitted with a single line of code, can eventually deliver a lot
+of value:
+
+```ruby
+include Gitlab::Events::Emittable
+
+emit Gitlab::Event.new('events/package/publish', package)
+```
+
+TODO A note about Presenters layer being used and being required to build a
+payload for an object being passed to the event.
+
+TODO Describe lack of a common schema in "Challenges" section.
+
+### Build a common way to route events through gateways
+
+TODO Describe an abstract gateway here. Provide examples around how to
+implement a `Gitlab::Events::Gateway::Base` abstract class.
+
+A gateway can subscribe to particular event groups. There is no need to
+subscribe to particular event, like it is required for `GitLab::EventStore`. If
+you create a subscription for a particular event type / event group, every
+additional `emit` implemented will not need code changes.
+
+### Integrate it all with existing `Gitlab::EventStore`
+
+Exisiting `Gitlab::EventStore` is an extension of what is proposed here, not a
+mutually exlusive solution, we do not intend to replace it.
+
+### Build a message queue outside of GitLab Rails
+
+At some point we may want to have a catch-all gateway that is going to push all
+events to an external system, that will become our events broker. This is out
+of scope of this blueprint, but should be mentioned and explained here.
+
+### Integrate GitLab CI Events
+
+Link to a blueprint that @furkanayhan is working on, describe the relationship,
+suggest a solution to build a gateway that will use subscriptions matching
+designed in [CI Workflows PoC](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91244).
+
+### Overview Diagram
+
+> I think that it might be a good idea to build a diagram that will visualize
+> the layers of abstraction that we want to implement this solution in. It is
+> very difficult to understand how id - for example - Kafka related to this
+> proposal. Simple explanation is that Kafka could be implemented in a
+> different layer of abstraction, and we could explain that better by building
+> a diagram.
+
+## Concepts
+
+### Hierarchical events
+
+Explain why we want to use `events/packages/publish` instead of `:publish_package_event`.
+
+Document opportunities related to using a wildcard matchers: `hooks/issues/*`.
+
+Describe a few event types and a few event groups with a few concrete events, provide examples.
+
+### Abstract gateways
+
+An overview of the purpose of `Gitlab::Events::Gateway::Base`, how gateways are
+supposed to work, how gateways decide what to do with events. Describe an idea
+to integrate existing system hooks with `emit` and refactor webhooks into a a
+Webhooks Gateway.
+
+### Persistence
+
+When do we want to persist an event? How is that relate to message queuing?
+
+### Event sourcing
+
+How can we levarage this system for event sourcing. Why we might want or want
+not to do that.
+
+### Event Brokers
+
+Explain the relationship with [Apache
+Kafka](https://en.wikipedia.org/wiki/Apache_Kafka) and explain why there is
+actually a very small overlap between what we want to achieve here and Kafka.
+
+### Cloud Events
+
+How can we implement `cloud/events/(source)/(event)` and make GitLab an event broker?
+
+## Principles
+
+1. Build a common abstraction for emitting events.
+1. Emitting an event should be a low-cost action, similarly to incrementing a Prometheus counter.
+1. Build a common abstraction for events' gateways.
+1. Integrate existing `Gitlab::EventStore`.
+1. If an event is supposed to go out of GitLab Rails, it needs to be stored.
+1. If it is not supposed to go out of GitLab Rails, a gateway can decide if it is going to be stored.
+1. Use hierarchical events, aggregated in event groups and then feature categories.
+
+## Way forward
+
+## What's next?
+
+
+## Status
+
+
+## Who
+
+
+<!-- vale gitlab.Spelling = NO -->
+
+| Role                         | Who
+|------------------------------|-------------------------|
+| Author                       | Grzegorz Bizon          |
+| Architecture Evolution Coach |                         |
+| Engineering Leader           |                         |
+| Product Manager              |                         |
+
+
+Domain experts:
+
+| Area                         | Who
+|------------------------------|------------------------|
+
+<!-- vale gitlab.Spelling = YES -->