Не подтверждена Коммит e1188d86 создал по автору Sean McGivern's avatar Sean McGivern
Просмотр файлов

Rewrite Sidekiq processes docs 

The existing docs focus too much on what you _can_ do, not what you
_should_ do. Let's reorder and rewrite them to fix that, with a clear
recommendation to not set up processes listening to specific queues.
владелец ddd78ac8
Скрыть пробелы
Построчно Рядом
doc/administration/geo/replication/tuning.md
Просмотр файла @ e1188d86
......@@ -30,7 +30,7 @@ However, this may not lead to more downloads in parallel unless the number of
available Sidekiq threads is also increased. For example, if repository synchronization
concurrency is increased from 25 to 50, you may also want to increase the number
of Sidekiq threads from 25 to 50. See the
[Sidekiq concurrency documentation](../../sidekiq/extra_sidekiq_processes.md#number-of-threads)
[Sidekiq concurrency documentation](../../sidekiq/extra_sidekiq_processes.md#concurrency)
for more details.
## Repository re-verification
......
doc/administration/operations/index.md
Просмотр файла @ e1188d86
......@@ -16,7 +16,6 @@ Keep your GitLab instance up and running smoothly.
- [Sidekiq MemoryKiller](sidekiq_memory_killer.md): Configure Sidekiq MemoryKiller
to restart Sidekiq.
- [Multiple Sidekiq processes](extra_sidekiq_processes.md): Configure multiple Sidekiq processes to ensure certain queues always have dedicated workers, no matter the number of jobs that must be processed. **(FREE SELF)**
- [Sidekiq routing rules](extra_sidekiq_routing.md): Configure the routing rules to route a job from a worker to a desirable queue. **(FREE SELF)**
- [Puma](puma.md): Understand Puma and puma-worker-killer.
- Speed up SSH operations by
[Authorizing SSH users via a fast, indexed lookup to the GitLab database](fast_ssh_key_lookup.md), and/or
......
doc/administration/sidekiq/extra_sidekiq_processes.md
Просмотр файла @ e1188d86
......@@ -6,91 +6,41 @@ info: To determine the technical writer assigned to the Stage/Group associated w
# Run multiple Sidekiq processes **(FREE SELF)**
GitLab allows you to start multiple Sidekiq processes.
These processes can be used to consume a dedicated set
of queues. This can be used to ensure certain queues always have dedicated
workers, no matter the number of jobs to be processed.
GitLab allows you to start multiple Sidekiq processes to process background jobs
at a higher rate on a single instance. By default, Sidekiq starts one worker
process and only uses a single core.
NOTE:
The information in this page applies only to Omnibus GitLab.
## Available Sidekiq queues
For a list of the existing Sidekiq queues, check the following files:
- [Queues for both GitLab Community and Enterprise Editions](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/workers/all_queues.yml)
- [Queues for GitLab Enterprise Editions only](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/workers/all_queues.yml)
Each entry in the above files represents a queue on which Sidekiq processes
can be started.
## Start multiple processes
> - [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/4006) in GitLab 12.10, starting multiple processes with Sidekiq cluster.
> - [Sidekiq cluster moved](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/181) to GitLab Free in 12.10.
> - [Sidekiq cluster became default](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/4140) in GitLab 13.0.
When starting multiple processes, the number of processes should
equal (and **not** exceed) the number of CPU cores you want to
dedicate to Sidekiq. Each Sidekiq process can use only 1 CPU
core, subject to the available workload and concurrency settings.
When starting multiple processes, the number of processes should at most equal
(and **not** exceed) the number of CPU cores you want to dedicate to Sidekiq.
The Sidekiq worker process uses no more than one CPU core.
To start multiple processes:
To start multiple processes, use the `sidekiq['queue_groups']` array setting to
specify how many processes to create using `sidekiq-cluster` and which queues
they should handle. Each item in the array equates to one additional Sidekiq
process, and values in each item determine the queues it works on. In the vast
majority of cases, all processes should listen to all queues (see
[processing specific job classes](processing_specific_job_classes.md) for more
details).
1. Using the `sidekiq['queue_groups']` array setting, specify how many processes to
create using `sidekiq-cluster` and which queue they should handle.
Each item in the array equates to one additional Sidekiq
process, and values in each item determine the queues it works on.
For example, to create four Sidekiq processes, each listening
to all available queues:
For example, the following setting creates three Sidekiq processes, one to run on
`elastic_commit_indexer`, one to run on `mailers`, and one process running on all queues:
1. Edit `/etc/gitlab/gitlab.rb`:
```ruby
sidekiq['queue_groups'] = [
"elastic_commit_indexer",
"mailers",
"*"
]
sidekiq['queue_groups'] = ['*'] * 4
```
To have an additional Sidekiq process handle multiple queues, add multiple
queue names to its item delimited by commas. For example:
```ruby
sidekiq['queue_groups'] = [
"elastic_commit_indexer, elastic_association_indexer",
"mailers",
"*"
]
```
[In GitLab 12.9](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/26594) and
later, the special queue name `*` means all queues. This starts two
processes, each handling all queues:
```ruby
sidekiq['queue_groups'] = [
"*",
"*"
]
```
`*` which matches all workers.
As a result, the wildcard query must stay at the end of the list or the rules after it are ignored.
`*` cannot be combined with concrete queue names - `*, mailers`
just handles the `mailers` queue.
When `sidekiq-cluster` is only running on a single node, make sure that at least
one process is running on all queues using `*`. This ensures a process
automatically picks up jobs in queues created in the future,
including queues that have dedicated processes.
If `sidekiq-cluster` is running on more than one node, you can also use
[`--negate`](#negate-settings) and list all the queues that are already being
processed.
1. Save the file and reconfigure GitLab for the changes to take effect:
1. Save the file and reconfigure GitLab:
```shell
sudo gitlab-ctl reconfigure
......@@ -101,125 +51,38 @@ To view the Sidekiq processes in GitLab:
1. On the top bar, select **Main menu > Admin**.
1. On the left sidebar, select **Monitoring > Background Jobs**.
## Negate settings
## Concurrency
To have the Sidekiq process work on every queue **except** the ones
you list. In this example, we exclude all import-related jobs from a Sidekiq node:
By default each process defined under `sidekiq` starts with a number of threads
that equals the number of queues, plus one spare thread, up to a maximum of 50.
For example, a process that handles all queues will use 50 threads by default.
1. Edit `/etc/gitlab/gitlab.rb` and add:
```ruby
sidekiq['negate'] = true
sidekiq['queue_selector'] = true
sidekiq['queue_groups'] = [
"feature_category=importers"
]
```
1. Save the file and reconfigure GitLab for the changes to take effect:
```shell
sudo gitlab-ctl reconfigure
```
## Queue selector
> - [Introduced](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/45) in GitLab 12.8.
> - [Sidekiq cluster, including queue selector, moved](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/181) to GitLab Free in 12.10.
> - [Renamed from `experimental_queue_selector` to `queue_selector`](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/147) in GitLab 13.6.
In addition to selecting queues by name, as above, the `queue_selector` option
allows queue groups to be selected in a more general way using a
[worker matching query](extra_sidekiq_routing.md#worker-matching-query). After `queue_selector`
is set, all `queue_groups` must follow the aforementioned syntax.
In `/etc/gitlab/gitlab.rb`:
```ruby
sidekiq['enable'] = true
sidekiq['queue_selector'] = true
sidekiq['queue_groups'] = [
# Run all non-CPU-bound queues that are high urgency
'resource_boundary!=cpu&urgency=high',
# Run all continuous integration and pages queues that are not high urgency
'feature_category=continuous_integration,pages&urgency!=high',
# Run all queues
'*'
]
```
## Ignore all import queues
When [importing from GitHub](../../user/project/import/github.md) or
other sources, Sidekiq might use all of its resources to perform those
operations. To set up two separate `sidekiq-cluster` processes, where
one only processes imports and the other processes all other queues:
1. Edit `/etc/gitlab/gitlab.rb` and add:
```ruby
sidekiq['enable'] = true
sidekiq['queue_selector'] = true
sidekiq['queue_groups'] = [
"feature_category=importers",
"feature_category!=importers"
]
```
1. Save the file and reconfigure GitLab for the changes to take effect:
```shell
sudo gitlab-ctl reconfigure
```
## Number of threads
By default each process defined under `sidekiq` starts with a
number of threads that equals the number of queues, plus one spare thread.
For example, a process that handles the `process_commit` and `post_receive`
queues uses three threads in total.
These thread run inside a single Ruby process, and each process
can only use a single CPU core. The usefulness of threading depends
on the work having some external dependencies to wait on, like database queries or
HTTP requests. Most Sidekiq deployments benefit from this threading, and when
running fewer queues in a process, increasing the thread count might be
even more desirable to make the most effective use of CPU resources.
These threads run inside a single Ruby process, and each process can only use a
single CPU core. The usefulness of threading depends on the work having some
external dependencies to wait on, like database queries or HTTP requests. Most
Sidekiq deployments benefit from this threading.
### Manage thread counts explicitly
The correct maximum thread count (also called concurrency) depends on the workload.
Typical values range from `1` for highly CPU-bound tasks to `15` or higher for mixed
low-priority work. A reasonable starting range is `15` to `25` for a non-specialized
deployment.
The correct maximum thread count (also called concurrency) depends on the
workload. Typical values range from `5` for highly CPU-bound tasks to `15` or
higher for mixed low-priority work. A reasonable starting range is `15` to `25`
for a non-specialized deployment.
You can find example values used by GitLab.com by searching for `concurrency:` in
[the Helm charts](https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/blob/master/releases/gitlab/values/gprd.yaml.gotmpl).
The values vary according to the work each specific deployment of Sidekiq does.
Any other specialized deployments with processes dedicated to specific queues should
have the concurrency tuned according to:
have the concurrency tuned according to:
We only recommend setting explicit concurrency by setting `min_concurrency` and
`max_concurrency` to the same value. The two values are kept for backwards
compatibility reasons, but for more predictable results, use the same value.
- The CPU usage of each type of process.
- The throughput achieved.
Each thread requires a Redis connection, so adding threads may increase Redis
latency and potentially cause client timeouts. See the
[Sidekiq documentation about Redis](https://github.com/mperham/sidekiq/wiki/Using-Redis) for more
details.
#### When running Sidekiq cluster (default)
For example, to set the concurrency to `20`:
Running Sidekiq cluster is the default in GitLab 13.0 and later.
1. Edit `/etc/gitlab/gitlab.rb` and add:
1. Edit `/etc/gitlab/gitlab.rb`:
```ruby
sidekiq['min_concurrency'] = 15
sidekiq['max_concurrency'] = 25
sidekiq['min_concurrency'] = 20
sidekiq['max_concurrency'] = 20
```
1. Save the file and reconfigure GitLab for the changes to take effect:
1. Save the file and reconfigure GitLab:
```shell
sudo gitlab-ctl reconfigure
......@@ -231,50 +94,45 @@ the other. Setting `min_concurrency` to `0` disables the limit.
For each queue group, let `N` be one more than the number of queues. The
concurrency is set to:
1. `min_concurrency`, if it's equal to `max_concurrency`.
1. `N`, if it's between `min_concurrency` and `max_concurrency`.
1. `max_concurrency`, if `N` exceeds this value.
1. `min_concurrency`, if `N` is less than this value.
If `min_concurrency` is equal to `max_concurrency`, then this value is used
regardless of the number of queues.
When `min_concurrency` is greater than `max_concurrency`, it is treated as
being equal to `max_concurrency`.
#### When running a single Sidekiq process
Running a single Sidekiq process is the default in GitLab 12.10 and earlier.
WARNING:
Running Sidekiq directly was removed in GitLab
[14.0](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/240).
1. Edit `/etc/gitlab/gitlab.rb` and add:
```ruby
sidekiq['cluster'] = false
sidekiq['concurrency'] = 25
```
1. Save the file and reconfigure GitLab for the changes to take effect:
You can find example values used by GitLab.com by searching for `concurrency:`
in [the Helm charts](https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/blob/master/releases/gitlab/values/gprd.yaml.gotmpl).
The values vary according to the work each specific deployment of Sidekiq does.
Any other specialized deployments with processes dedicated to specific queues
should have the concurrency tuned according to:
```shell
sudo gitlab-ctl reconfigure
```
- The CPU usage of each type of process.
- The throughput achieved.
This sets the concurrency (number of threads) for the Sidekiq process.
Each thread requires a Redis connection, so adding threads may increase Redis
latency and potentially cause client timeouts. See the [Sidekiq documentation about Redis](https://github.com/mperham/sidekiq/wiki/Using-Redis)
for more details.
## Modify the check interval
To modify `sidekiq-cluster`'s health check interval for the additional Sidekiq processes:
To modify Sidekiq's health check interval for the additional Sidekiq
processes:
1. Edit `/etc/gitlab/gitlab.rb` and add (the value can be any integer number of seconds):
1. Edit `/etc/gitlab/gitlab.rb`:
```ruby
sidekiq['interval'] = 5
```
1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
The value can be any integer number of seconds.
1. Save the file and reconfigure GitLab:
```shell
sudo gitlab-ctl reconfigure
```
## Troubleshoot using the CLI
......@@ -291,6 +149,9 @@ takes arguments using the following syntax:
/opt/gitlab/embedded/service/gitlab-rails/bin/sidekiq-cluster [QUEUE,QUEUE,...] [QUEUE, ...]
```
The `--dryrun` argument allows viewing the command to be executed without
actually starting it.
Each separate argument denotes a group of queues that have to be processed by a
Sidekiq process. Multiple queues can be processed by the same process by
separating them with a comma instead of a space.
......@@ -301,29 +162,6 @@ explicitly list all the queue names. For more information about queue namespaces
see the relevant section in the
[Sidekiq development documentation](../../development/sidekiq/index.md#queue-namespaces).
For example, say you want to start 2 extra processes: one to process the
`process_commit` queue, and one to process the `post_receive` queue. This can be
done as follows:
```shell
/opt/gitlab/embedded/service/gitlab-rails/bin/sidekiq-cluster process_commit post_receive
```
If you instead want to start one process processing both queues, you'd use the
following syntax:
```shell
/opt/gitlab/embedded/service/gitlab-rails/bin/sidekiq-cluster process_commit,post_receive
```
If you want to have one Sidekiq process dealing with the `process_commit` and
`post_receive` queues, and one process to process the `gitlab_shell` queue,
you'd use the following:
```shell
/opt/gitlab/embedded/service/gitlab-rails/bin/sidekiq-cluster process_commit,post_receive gitlab_shell
```
### Monitor the `sidekiq-cluster` command
The `sidekiq-cluster` command does not terminate once it has started the desired
......
doc/administration/sidekiq/extra_sidekiq_routing.md
Просмотр файла @ e1188d86
---
stage: Systems
group: Distribution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
redirect_to: 'processing_specific_job_classes.md#routing-rules'
remove_date: '2023-02-01'
---
# Queue routing rules **(FREE SELF)**
This document was moved to [another location](processing_specific_job_classes.md#routing-rules).
When the number of Sidekiq jobs increases to a certain scale, the system faces
some scalability issues. One of them is that the length of the queue tends to get
longer. High-urgency jobs have to wait longer until other less urgent jobs
finish. This head-of-line blocking situation may eventually affect the
responsiveness of the system, especially critical actions. In another scenario,
the performance of some jobs is degraded due to other long running or CPU-intensive jobs
(computing or rendering ones) in the same machine.
To counter the aforementioned issues, one effective solution is to split
Sidekiq jobs into different queues and assign machines handling each queue
exclusively. For example, all CPU-intensive jobs could be routed to the
`cpu-bound` queue and handled by a fleet of CPU optimized instances. The queue
topology differs between companies depending on the workloads and usage
patterns. Therefore, GitLab supports a flexible mechanism for the
administrator to route the jobs based on their characteristics.
As an alternative to [Queue selector](extra_sidekiq_processes.md#queue-selector), which
configures Sidekiq cluster to listen to a specific set of workers or queues,
GitLab also supports routing a job from a worker to the desired queue when it
is scheduled. Sidekiq clients try to match a job against a configured list of
routing rules. Rules are evaluated from first to last, and as soon as we find a
match for a given worker we stop processing for that worker (first match wins).
If the worker doesn't match any rule, it falls back to the queue name generated
from the worker name.
By default, if the routing rules are not configured (or denoted with an empty
array), all the jobs are routed to the queue generated from the worker name.
## Example configuration
In `/etc/gitlab/gitlab.rb`:
```ruby
sidekiq['routing_rules'] = [
# Do not re-route workers that require their own queue
['tags=needs_own_queue', nil],
# Route all non-CPU-bound workers that are high urgency to `high-urgency` queue
['resource_boundary!=cpu&urgency=high', 'high-urgency'],
# Route all database, gitaly and global search workers that are throttled to `throttled` queue
['feature_category=database,gitaly,global_search&urgency=throttled', 'throttled'],
# Route all workers having contact with outside work to a `network-intenstive` queue
['has_external_dependencies=true|feature_category=hooks|tags=network', 'network-intensive'],
# Route all import workers to the queues generated by the worker name, for
# example, JiraImportWorker to `jira_import`, SVNWorker to `svn_worker`
['feature_category=import', nil],
# Wildcard matching, route the rest to `default` queue
['*', 'default']
]
```
The routing rules list is an order-matter array of tuples of query and
corresponding queue:
- The query is following a [worker matching query](#worker-matching-query) syntax.
- The `<queue_name>` must be a valid Sidekiq queue name. If the queue name
is `nil`, or an empty string, the worker is routed to the queue generated
by the name of the worker instead.
The query supports wildcard matching `*`, which matches all workers. As a
result, the wildcard query must stay at the end of the list or the rules after it
are ignored.
NOTE:
Mixing queue routing rules and queue selectors requires care to
ensure all jobs that are scheduled and picked up by appropriate Sidekiq
workers.
## Worker matching query
GitLab provides a query syntax to match a worker based on its
attributes. This query syntax is employed by both
[Queue routing rules](#queue-routing-rules) and
[Queue selector](extra_sidekiq_processes.md#queue-selector). A query includes two
components:
- Attributes that can be selected.
- Operators used to construct a query.
### Available attributes
> [Introduced](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/261) in GitLab 13.1 (`tags`).
Queue matching query works upon the worker attributes, described in
[Sidekiq style guide](../../development/sidekiq/index.md). We support querying
based on a subset of worker attributes:
- `feature_category` - the
[GitLab feature category](https://about.gitlab.com/direction/maturity/#category-maturity) the
queue belongs to. For example, the `merge` queue belongs to the
`source_code_management` category.
- `has_external_dependencies` - whether or not the queue connects to external
services. For example, all importers have this set to `true`.
- `urgency` - how important it is that this queue's jobs run
quickly. Can be `high`, `low`, or `throttled`. For example, the
`authorized_projects` queue is used to refresh user permissions, and
is `high` urgency.
- `worker_name` - the worker name. Use this attribute to select a specific worker.
- `name` - the queue name generated from the worker name. Use this attribute to select a specific queue. Because this is generated from
the worker name, it does not change based on the result of other routing
rules.
- `resource_boundary` - if the queue is bound by `cpu`, `memory`, or
`unknown`. For example, the `ProjectExportWorker` is memory bound as it has
to load data in memory before saving it for export.
- `tags` - short-lived annotations for queues. These are expected to frequently
change from release to release, and may be removed entirely.
`has_external_dependencies` is a boolean attribute: only the exact
string `true` is considered true, and everything else is considered
false.
`tags` is a set, which means that `=` checks for intersecting sets, and
`!=` checks for disjoint sets. For example, `tags=a,b` selects queues
that have tags `a`, `b`, or both. `tags!=a,b` selects queues that have
neither of those tags.
The attributes of each worker are hard-coded in the source code. For
convenience, we generate a
[list of all available attributes in GitLab Community Edition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/workers/all_queues.yml)
and a
[list of all available attributes in GitLab Enterprise Edition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/workers/all_queues.yml).
### Available operators
`queue_selector` supports the following operators, listed from highest
to lowest precedence:
- `|` - the logical `OR` operator. For example, `query_a|query_b` (where `query_a`
and `query_b` are queries made up of the other operators here) includes
queues that match either query.
- `&` - the logical `AND` operator. For example, `query_a&query_b` (where
`query_a` and `query_b` are queries made up of the other operators here) will
only include queues that match both queries.
- `!=` - the `NOT IN` operator. For example, `feature_category!=issue_tracking`
excludes all queues from the `issue_tracking` feature category.
- `=` - the `IN` operator. For example, `resource_boundary=cpu` includes all
queues that are CPU bound.
- `,` - the concatenate set operator. For example,
`feature_category=continuous_integration,pages` includes all queues from
either the `continuous_integration` category or the `pages` category. This
example is also possible using the OR operator, but allows greater brevity, as
well as being lower precedence.
The operator precedence for this syntax is fixed: it's not possible to make `AND`
have higher precedence than `OR`.
[In GitLab 12.9](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/26594) and
later, as with the standard queue group syntax above, a single `*` as the
entire queue group selects all queues.
### Migration
After the Sidekiq routing rules are changed, administrators must take care
with the migration to avoid losing jobs entirely, especially in a system with
long queues of jobs. The migration can be done by following the migration steps
mentioned in [Sidekiq job migration](sidekiq_job_migration.md)
<!-- This redirect file can be deleted after <2023-02-01>. -->
<!-- Redirects that point to other docs in the same project expire in three months. -->
<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. -->
<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html -->
doc/administration/sidekiq/index.md
Просмотр файла @ e1188d86
......@@ -398,7 +398,7 @@ sudo gitlab-rake gitlab:features:disable_rugged
## Related topics
- [Extra Sidekiq processes](extra_sidekiq_processes.md)
- [Extra Sidekiq routing](extra_sidekiq_routing.md)
- [Processing specific job classes](processing_specific_job_classes.md)
- [Sidekiq health checks](sidekiq_health_check.md)
- [Using the GitLab-Sidekiq chart](https://docs.gitlab.com/charts/charts/gitlab/sidekiq/)
......
doc/administration/sidekiq/processing_specific_job_classes.md 0 → 100644
Просмотр файла @ e1188d86
---
stage: Systems
group: Distribution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
# Processing specific job classes
WARNING:
These are advanced settings. While they are used on GitLab.com, most GitLab
instances should add more processes that all listen to all queues. This is the
same approach we take in our [Reference Architectures](../reference_architectures/index.md).
GitLab has two options for creating Sidekiq processes that only handle specific
job classes:
1. [Routing rules](#routing-rules) are used on GitLab.com. They direct jobs
inside the application to queue names configured by administrators. This
lowers the load on Redis, which is important on very large-scale deployments.
1. [Queue selectors](#queue-selectors) perform the job selection outside the
application, when starting the Sidekiq process. This was used on GitLab.com
until September 2021, and is retained for compatibility reasons.
Both of these use the same [worker matching query](#worker-matching-query)
syntax. While they can technically be used together, most deployments should
choose one or the other; there is no particular benefit in combining them.
Routing rules must be the same across all GitLab nodes as they are part of the
application configuration. Queue selectors can be different across GitLab nodes
because they only change the arguments to the launched Sidekiq process.
## Routing rules
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/59604) in GitLab 13.12.
> - [Default routing rule value](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/97908) added in GitLab 15.4.
We recommend most GitLab instances using routing rules to manage their Sidekiq
queues. This allows administrators to choose single queue names for groups of
job classes based on their attributes. The syntax is an ordered array of pairs of `[query, queue]`:
1. The query is a [worker matching query](#worker-matching-query).
1. The queue name must be a valid Sidekiq queue name. If the queue name
is `nil`, or an empty string, the worker is routed to the queue generated
by the name of the worker instead. (See [list of available job classes](#list-of-available-job-classes)
for more information).
The queue name does not have to match any existing queue name in the
list of available job classes.
1. The first query matching a worker is chosen for that worker; later rules are
ignored.
### Routing rules migration
After the Sidekiq routing rules are changed, administrators must take care with
the migration to avoid losing jobs entirely, especially in a system with long
queues of jobs. The migration can be done by following the migration steps
mentioned in [Sidekiq job migration](sidekiq_job_migration.md).
### Detailed example
This is a comprehensive example intended to show different possibilities. It is
not a recommendation.
1. Edit `/etc/gitlab/gitlab.rb`:
```ruby
sidekiq['routing_rules'] = [
# Route all non-CPU-bound workers that are high urgency to `high-urgency` queue
['resource_boundary!=cpu&urgency=high', 'high-urgency'],
# Route all database, gitaly and global search workers that are throttled to `throttled` queue
['feature_category=database,gitaly,global_search&urgency=throttled', 'throttled'],
# Route all workers having contact with outside world to a `network-intenstive` queue
['has_external_dependencies=true|feature_category=hooks|tags=network', 'network-intensive'],
# Route all import workers to the queues generated by the worker name, for
# example, JiraImportWorker to `jira_import`, SVNWorker to `svn_worker`
['feature_category=import', 'import'],
# Wildcard matching, route the rest to `default` queue
['*', 'default']
]
```
The `queue_groups` can then be set to match these generated queue names. For
instance:
```ruby
sidekiq['queue_selector'] = false
sidekiq['queue_groups'] = [
# Run two high-urgency processes
'high-urgency',
'high-urgency',
# Run one process for throttled, network-intensive, import
'throttled,network-intensive,import',
# Run one 'catchall' process on the default queue
'default'
]
```
1. Save the file and reconfigure GitLab:
```shell
sudo gitlab-ctl reconfigure
```
## Queue selectors
> - [Introduced](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/45) in GitLab 12.8.
> - [Sidekiq cluster, including queue selector, moved](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/181) to GitLab Free in 12.10.
> - [Renamed from `experimental_queue_selector` to `queue_selector`](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/147) in GitLab 13.6.
The `queue_selector` option allows queue groups to be selected in a more general
way using a [worker matching query](#worker-matching-query). After
`queue_selector` is set, all `queue_groups` must follow the aforementioned
syntax.
### Using queue selectors
1. Edit `/etc/gitlab/gitlab.rb`:
```ruby
sidekiq['enable'] = true
sidekiq['routing_rules'] = [['*', nil]]
sidekiq['queue_selector'] = true
sidekiq['queue_groups'] = [
# Run all non-CPU-bound queues that are high urgency
'resource_boundary!=cpu&urgency=high',
# Run all continuous integration and pages queues that are not high urgency
'feature_category=continuous_integration,pages&urgency!=high',
# Run all queues
'*'
]
```
1. Save the file and reconfigure GitLab:
```shell
sudo gitlab-ctl reconfigure
```
### Negate settings
This allows you to have the Sidekiq process work on every queue **except** the
ones you list. This is generally only used when there are multiple Sidekiq
nodes. In this example, we exclude all import-related jobs from a Sidekiq node.
1. Edit `/etc/gitlab/gitlab.rb`:
```ruby
sidekiq['routing_rules'] = [['*', nil]]
sidekiq['negate'] = true
sidekiq['queue_selector'] = true
sidekiq['queue_groups'] = [
"feature_category=importers"
]
```
1. Save the file and reconfigure GitLab:
```shell
sudo gitlab-ctl reconfigure
```
## Worker matching query
GitLab provides a query syntax to match a worker based on its attributes. This
query syntax is employed by both [routing rules](#routing-rules) and
[queue selectors](#queue-selectors). A query includes two components:
- Attributes that can be selected.
- Operators used to construct a query.
### Available attributes
> [Introduced](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/261) in GitLab 13.1 (`tags`).
Queue matching query works upon the worker attributes, described in
[Sidekiq style guide](../../development/sidekiq/index.md). We support querying
based on a subset of worker attributes:
- `feature_category` - the
[GitLab feature category](https://about.gitlab.com/direction/maturity/#category-maturity) the
queue belongs to. For example, the `merge` queue belongs to the
`source_code_management` category.
- `has_external_dependencies` - whether or not the queue connects to external
services. For example, all importers have this set to `true`.
- `urgency` - how important it is that this queue's jobs run
quickly. Can be `high`, `low`, or `throttled`. For example, the
`authorized_projects` queue is used to refresh user permissions, and
is `high` urgency.
- `worker_name` - the worker name. Use this attribute to select a specific worker. Find all available names in [the job classes lists](#list-of-available-job-classes) below.
- `name` - the queue name generated from the worker name. Use this attribute to select a specific queue. Because this is generated from
the worker name, it does not change based on the result of other routing
rules.
- `resource_boundary` - if the queue is bound by `cpu`, `memory`, or
`unknown`. For example, the `ProjectExportWorker` is memory bound as it has
to load data in memory before saving it for export.
- `tags` - short-lived annotations for queues. These are expected to frequently
change from release to release, and may be removed entirely.
`has_external_dependencies` is a boolean attribute: only the exact
string `true` is considered true, and everything else is considered
false.
`tags` is a set, which means that `=` checks for intersecting sets, and
`!=` checks for disjoint sets. For example, `tags=a,b` selects queues
that have tags `a`, `b`, or both. `tags!=a,b` selects queues that have
neither of those tags.
### Available operators
Routing rules and queue selectors support the following operators, listed from
highest to lowest precedence:
- `|` - the logical `OR` operator. For example, `query_a|query_b` (where `query_a`
and `query_b` are queries made up of the other operators here) includes
queues that match either query.
- `&` - the logical `AND` operator. For example, `query_a&query_b` (where
`query_a` and `query_b` are queries made up of the other operators here) will
only include queues that match both queries.
- `!=` - the `NOT IN` operator. For example, `feature_category!=issue_tracking`
excludes all queues from the `issue_tracking` feature category.
- `=` - the `IN` operator. For example, `resource_boundary=cpu` includes all
queues that are CPU bound.
- `,` - the concatenate set operator. For example,
`feature_category=continuous_integration,pages` includes all queues from
either the `continuous_integration` category or the `pages` category. This
example is also possible using the OR operator, but allows greater brevity, as
well as being lower precedence.
The operator precedence for this syntax is fixed: it's not possible to make `AND`
have higher precedence than `OR`.
As with the standard queue group syntax above, a single `*` as the
entire queue group selects all queues.
### List of available job classes
For a list of the existing Sidekiq job classes and queues, check the following
files:
- [Queues for all GitLab editions](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/workers/all_queues.yml)
- [Queues for GitLab Enterprise Editions only](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/workers/all_queues.yml)
doc/development/sidekiq/index.md
Просмотр файла @ e1188d86
......@@ -27,7 +27,7 @@ There are pages with additional detail on the following topics:
All workers should include `ApplicationWorker` instead of `Sidekiq::Worker`,
which adds some convenience methods and automatically sets the queue based on
the [routing rules](../../administration/sidekiq/extra_sidekiq_routing.md#queue-routing-rules).
the [routing rules](../../administration/sidekiq/processing_specific_job_classes.md#routing-rules).
## Retries
......@@ -88,7 +88,7 @@ error rate.
Previously, each worker had its own queue, which was automatically set based on the
worker class name. For a worker named `ProcessSomethingWorker`, the queue name
would be `process_something`. You can now route workers to a specific queue using
[queue routing rules](../../administration/sidekiq/extra_sidekiq_routing.md#queue-routing-rules).
[queue routing rules](../../administration/sidekiq/processing_specific_job_classes.md#routing-rules).
In GDK, new workers are routed to a queue named `default`.
If you're not sure what queue a worker uses,
......
doc/integration/advanced_search/elasticsearch.md
Просмотр файла @ e1188d86
......@@ -783,8 +783,8 @@ additional process dedicated to indexing a set of queues (or queue group). This
ensure that indexing queues always have a dedicated worker, while the rest of the queues have
another dedicated worker to avoid contention.
For this purpose, use the [queue selector](../../administration/sidekiq/extra_sidekiq_processes.md#queue-selector)
option that allows a more general selection of queue groups using a [worker matching query](../../administration/sidekiq/extra_sidekiq_routing.md#worker-matching-query).
For this purpose, use the [queue selectors](../../administration/sidekiq/processing_specific_job_classes.md#queue-selectors)
option that allows a more general selection of queue groups using a [worker matching query](../../administration/sidekiq/processing_specific_job_classes.md#worker-matching-query).
To handle these two queue groups, we generally recommend one of the following two options. You can either:
......@@ -804,8 +804,8 @@ To create both an indexing and a non-indexing Sidekiq process in one node:
```ruby
sidekiq['enable'] = true
sidekiq['queue_selector'] = true
sidekiq['queue_groups'] = [
sidekiq['queue_selector'] = true
sidekiq['queue_groups'] = [
"feature_category=global_search",
"feature_category!=global_search"
]
......
doc/integration/advanced_search/elasticsearch_troubleshooting.md
Просмотр файла @ e1188d86
......@@ -248,7 +248,7 @@ sudo gitlab-rake gitlab:elastic:clear_locked_projects
If `ElasticCommitIndexerWorker` Sidekiq workers are failing with this error during indexing, it usually means that Elasticsearch is unable to keep up with the concurrency of indexing request. To address change the following settings:
- To decrease the indexing throughput you can decrease `Bulk request concurrency` (see [Advanced Search settings](elasticsearch.md#advanced-search-configuration)). This is set to `10` by default, but you change it to as low as 1 to reduce the number of concurrent indexing operations.
- If changing `Bulk request concurrency` didn't help, you can use the [queue selector](../../administration/sidekiq/extra_sidekiq_processes.md#queue-selector) option to [limit indexing jobs only to specific Sidekiq nodes](elasticsearch.md#index-large-instances-with-dedicated-sidekiq-nodes-or-processes), which should reduce the number of indexing requests.
- If changing `Bulk request concurrency` didn't help, you can use the [queue selector](../../administration/sidekiq/processing_specific_job_classes.md#queue-selectors) option to [limit indexing jobs only to specific Sidekiq nodes](elasticsearch.md#index-large-instances-with-dedicated-sidekiq-nodes-or-processes), which should reduce the number of indexing requests.
### Indexing is very slow or fails with `rejected execution of coordinating operation` messages
......
doc/update/index.md
Просмотр файла @ e1188d86
......@@ -477,7 +477,7 @@ and [Helm Chart deployments](https://docs.gitlab.com/charts/). They come with ap
### 15.5.0
- GitLab 15.4.0 introduced a default [Sidekiq routing rule](../administration/sidekiq/extra_sidekiq_routing.md) that routes all jobs to the `default` queue. For instances using [queue selectors](../administration/sidekiq/extra_sidekiq_processes.md#queue-selector), this will cause [performance problems](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1991) as some Sidekiq processes will be idle.
- GitLab 15.4.0 introduced a default [Sidekiq routing rule](../administration/sidekiq/extra_sidekiq_routing.md) that routes all jobs to the `default` queue. For instances using [queue selectors](../administration/sidekiq/processing_specific_job_classes.md#queue-selectors), this will cause [performance problems](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1991) as some Sidekiq processes will be idle.
- The default routing rule has been reverted in 15.5.4, so upgrading to that version or later will return to the previous behavior.
- If a GitLab instance now listens only to the `default` queue (which is not currently recommended), it will be required to add this routing rule back in `/etc/gitlab/gitlab.rb`:
......@@ -490,7 +490,7 @@ and [Helm Chart deployments](https://docs.gitlab.com/charts/). They come with ap
- GitLab 15.4.0 includes a [batched background migration](#batched-background-migrations) to [remove incorrect values from `expire_at` in `ci_job_artifacts` table](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/89318).
This migration might take hours or days to complete on larger GitLab instances.
- By default, Gitaly and Praefect nodes use the time server at `pool.ntp.org`. If your instance can not connect to `pool.ntp.org`, [configure the `NTP_HOST` variable](../administration/gitaly/praefect.md#customize-time-server-setting).
- GitLab 15.4.0 introduced a default [Sidekiq routing rule](../administration/sidekiq/extra_sidekiq_routing.md) that routes all jobs to the `default` queue. For instances using [queue selectors](../administration/sidekiq/extra_sidekiq_processes.md#queue-selector), this will cause [performance problems](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1991) as some Sidekiq processes will be idle.
- GitLab 15.4.0 introduced a default [Sidekiq routing rule](../administration/sidekiq/extra_sidekiq_routing.md) that routes all jobs to the `default` queue. For instances using [queue selectors](../administration/sidekiq/processing_specific_job_classes.md#queue-selectors), this will cause [performance problems](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1991) as some Sidekiq processes will be idle.
- The default routing rule has been reverted in 15.4.5, so upgrading to that version or later will return to the previous behavior.
- If a GitLab instance now listens only to the `default` queue (which is not currently recommended), it will be required to add this routing rule back in `/etc/gitlab/gitlab.rb`:
......
Поддерживает Markdown
0% или .
You are about to add 0 people to the discussion. Proceed with caution.
Сначала завершите редактирование этого сообщения!
Пожалуйста, зарегистрируйтесь или чтобы прокомментировать