Коммит 36084610 создал по автору Marcel Amirault's avatar Marcel Amirault
Просмотр файлов

Move 4 testing-related docs to ci/testing 

Moves fail fast testing, load performance testing,
test coverage visualization and metrics reports to
ci/testing directory
владелец cb549e15
Скрыть пробелы
Построчно Рядом
doc/ci/examples/index.md
Просмотр файла @ 36084610
......@@ -28,7 +28,7 @@ The following table lists examples with step-by-step tutorials that are containe
| Deployment with Dpl | [Using `dpl` as deployment tool](deployment/index.md). |
| GitLab Pages | See the [GitLab Pages](../../user/project/pages/index.md) documentation for a complete example of deploying a static site. |
| End-to-end testing | [End-to-end testing with GitLab CI/CD and WebdriverIO](end_to_end_testing_webdriverio/index.md). |
| Load performance testing | [Load Performance Testing with the k6 container](../../user/project/merge_requests/load_performance_testing.md). |
| Load performance testing | [Load Performance Testing with the k6 container](../testing/load_performance_testing.md). |
| Multi project pipeline | [Build, test deploy using multi project pipeline](https://gitlab.com/gitlab-examples/upstream-project). |
| npm with semantic-release | [Publish npm packages to the GitLab Package Registry using semantic-release](semantic-release.md). |
| PHP with Laravel, Envoy | [Test and deploy Laravel applications with GitLab CI/CD and Envoy](laravel_with_gitlab_and_envoy/index.md). |
......
doc/ci/index.md
Просмотр файла @ 36084610
......@@ -84,7 +84,7 @@ GitLab CI/CD features, grouped by DevOps stage, include:
| [Connect to cloud services](cloud_services/index.md) | Connect to cloud providers using OpenID Connect (OIDC) to retrieve temporary credentials to access services or secrets. |
| **Verify** | |
| [Browser Performance Testing](testing/browser_performance_testing.md) | Quickly determine the browser performance impact of pending code changes. |
| [Load Performance Testing](../user/project/merge_requests/load_performance_testing.md) | Quickly determine the server performance impact of pending code changes. |
| [Load Performance Testing](testing/load_performance_testing.md) | Quickly determine the server performance impact of pending code changes. |
| [CI services](services/index.md) | Link Docker containers with your base image. |
| [GitLab CI/CD for external repositories](ci_cd_for_external_repos/index.md) | Get the benefits of GitLab CI/CD combined with repositories in GitHub and Bitbucket Cloud. |
| [Interactive Web Terminals](interactive_web_terminal/index.md) | Open an interactive web terminal to debug the running jobs. |
......
doc/ci/metrics_reports.md
Просмотр файла @ 36084610
---
stage: Verify
group: Pipeline Insights
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
redirect_to: 'testing/metrics_reports.md'
remove_date: '2022-08-31'
---
# Metrics Reports **(PREMIUM)**
This document was moved to [another location](testing/metrics_reports.md).
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9788) in GitLab 11.10. Requires GitLab Runner 11.10 and above.
GitLab provides a lot of great reporting tools for things like [merge requests](../user/project/merge_requests/index.md) - [Unit test reports](testing/unit_test_reports.md), [code quality](testing/code_quality.md), and performance tests. While JUnit is a great open framework for tests that "pass" or "fail", it is also important to see other types of metrics from a given change.
You can configure your job to use custom Metrics Reports, and GitLab displays a report on the merge request so that it's easier and faster to identify changes without having to check the entire log.
![Metrics Reports](img/metrics_reports_v13_0.png)
## Use cases
Consider the following examples of data that can use Metrics Reports:
1. Memory usage
1. Load testing results
1. Code complexity
1. Code coverage stats
## How it works
Metrics for a branch are read from the latest metrics report artifact (default filename: `metrics.txt`) as string values.
For an MR, the values of these metrics from the feature branch are compared to the values from the target branch. Then they are displayed in the MR widget in this order:
- Existing metrics with changed values.
- Metrics that have been added by the MR. Marked with a **New** badge.
- Metrics that have been removed by the MR. Marked with a **Removed** badge.
- Existing metrics with unchanged values.
## How to set it up
Add a job that creates a [metrics report](yaml/artifacts_reports.md#artifactsreportsmetrics) (default filename: `metrics.txt`). The file should conform to the [OpenMetrics](https://openmetrics.io/) format.
For example:
```yaml
metrics:
script:
- echo 'metric_name metric_value' > metrics.txt
artifacts:
reports:
metrics: metrics.txt
```
## Advanced Example
An advanced example of an OpenMetrics text file (from the [Prometheus documentation](https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exposition_formats.md#text-format-example))
renders in the merge request widget as:
![Metrics Reports Advanced](img/metrics_reports_advanced_v13_0.png)
## Troubleshooting
### Metrics reports did not change
You can see `Metrics reports did not change` when trying to view metrics reports in merge requests. Reasons for this are:
- The target branch for the merge request doesn't have a baseline metrics report for comparison.
- You don't have GitLab license at the Premium tier or above.
There is [an issue open](https://gitlab.com/gitlab-org/gitlab/-/issues/343065) to improve this message.
<!-- This redirect file can be deleted after <2022-09-22>. -->
<!-- Redirects that point to other docs in the same project expire in three months. -->
<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. -->
<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html -->
doc/ci/pipelines/pipeline_artifacts.md
Просмотр файла @ 36084610
......@@ -10,7 +10,7 @@ Pipeline artifacts are files created by GitLab after a pipeline finishes. Pipeli
different to [job artifacts](job_artifacts.md) because they are not explicitly managed by
`.gitlab-ci.yml` definitions.
Pipeline artifacts are used by the [test coverage visualization feature](../../user/project/merge_requests/test_coverage_visualization.md)
Pipeline artifacts are used by the [test coverage visualization feature](../testing/test_coverage_visualization.md)
to collect coverage information.
## Storage
......
doc/ci/pipelines/pipeline_efficiency.md
Просмотр файла @ 36084610
......@@ -174,7 +174,7 @@ to stop them from running:
Ensure that errors are detected early in the CI/CD pipeline. A job that takes a very long
time to complete keeps a pipeline from returning a failed status until the job completes.
Design pipelines so that jobs that can [fail fast](../../user/project/merge_requests/fail_fast_testing.md)
Design pipelines so that jobs that can [fail fast](../testing/fail_fast_testing.md)
run earlier. For example, add an early stage and move the syntax, style linting,
Git commit message verification, and similar jobs in there.
......
doc/ci/testing/fail_fast_testing.md 0 → 100644
Просмотр файла @ 36084610
---
stage: Verify
group: Pipeline Insights
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
---
# Fail Fast Testing **(PREMIUM)**
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/198550) in GitLab 13.1.
For applications that use RSpec for running tests, we've introduced the `Verify/Failfast`
[template to run subsets of your test suite](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/templates/Verify/FailFast.gitlab-ci.yml),
based on the changes in your merge request.
The template uses the [test_file_finder (`tff`) gem](https://gitlab.com/gitlab-org/ci-cd/test_file_finder/)
that accepts a list of files as input, and returns a list of spec (test) files
that it believes to be relevant to the input files.
`tff` is designed for Ruby on Rails projects, so the `Verify/FailFast` template is
configured to run when changes to Ruby files are detected. By default, it runs in
the [`.pre` stage](../yaml/index.md#stage-pre) of a GitLab CI/CD pipeline,
before all other stages.
## Example use case
Fail fast testing is useful when adding new functionality to a project and adding
new automated tests.
Your project could have hundreds of thousands of tests that take a long time to complete.
You may be confident that a new test will pass, but you have to wait for all the tests
to complete to verify it. This could take an hour or more, even when using parallelization.
Fail fast testing gives you a faster feedback loop from the pipeline. It lets you
know quickly that the new tests are passing and the new functionality did not break
other tests.
## Requirements
This template requires:
- A project built in Rails that uses RSpec for testing.
- CI/CD configured to:
- Use a Docker image with Ruby available.
- Use [Merge request pipelines](../pipelines/merge_request_pipelines.md#prerequisites)
- [Merged results pipelines](../pipelines/merged_results_pipelines.md#enable-merged-results-pipelines)
enabled in the project settings.
- A Docker image with Ruby available. The template uses `image: ruby:2.6` by default, but you [can override](../yaml/includes.md#override-included-configuration-values) this.
## Configuring Fast RSpec Failure
We use the following plain RSpec configuration as a starting point. It installs all the
project gems and executes `rspec`, on merge request pipelines only.
```yaml
rspec-complete:
stage: test
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
script:
- bundle install
- bundle exec rspec
```
To run the most relevant specs first instead of the whole suite, [`include`](../yaml/index.md#include)
the template by adding the following to your CI/CD configuration:
```yaml
include:
- template: Verify/FailFast.gitlab-ci.yml
```
To customize the job, specific options may be set to override the template. For example, to override the default Docker image:
```yaml
include:
- template: Verify/FailFast.gitlab-ci.yml
rspec-rails-modified-path-specs:
image: custom-docker-image-with-ruby
```
### Example test loads
For illustrative purposes, let's say our Rails app spec suite consists of 100 specs per model for ten models.
If no Ruby files are changed:
- `rspec-rails-modified-paths-specs` does not run any tests.
- `rspec-complete` runs the full suite of 1000 tests.
If one Ruby model is changed, for example `app/models/example.rb`, then `rspec-rails-modified-paths-specs`
runs the 100 tests for `example.rb`:
- If all of these 100 tests pass, then the full `rspec-complete` suite of 1000 tests is allowed to run.
- If any of these 100 tests fail, they fail quickly, and `rspec-complete` does not run any tests.
The final case saves resources and time as the full 1000 test suite does not run.
doc/ci/testing/index.md
Просмотр файла @ 36084610
......@@ -9,18 +9,18 @@ info: To determine the technical writer assigned to the Stage/Group associated w
Use GitLab CI/CD to test the changes included in a feature branch. You can also
display reports or link to important information directly from [merge requests](../../user/project/merge_requests/index.md).
| Feature | Description |
|-------------------------------------------------------------------------------------------------|-------------|
| [Accessibility Testing](accessibility_testing.md) | Automatically report A11y violations for changed pages in merge requests. |
| [Browser Performance Testing](browser_performance_testing.md) | Quickly determine the browser performance impact of pending code changes. |
| [Load Performance Testing](../../user/project/merge_requests/load_performance_testing.md) | Quickly determine the server performance impact of pending code changes. |
| [Code Quality](code_quality.md) | Analyze your source code quality using the [Code Climate](https://codeclimate.com/) analyzer and show the Code Climate report right in the merge request widget area. |
| [Display arbitrary job artifacts](../yaml/index.md#artifactsexpose_as) | Configure CI pipelines with the `artifacts:expose_as` parameter to directly link to selected [artifacts](../pipelines/job_artifacts.md) in merge requests. |
| [Unit test reports](unit_test_reports.md) | Configure your CI jobs to use Unit test reports, and let GitLab display a report on the merge request so that it's easier and faster to identify the failure without having to check the entire job log. |
| [License Compliance](../../user/compliance/license_compliance/index.md) | Manage the licenses of your dependencies. |
| [Metrics Reports](../metrics_reports.md) | Display the Metrics Report on the merge request so that it's fast and easier to identify changes to important metrics. |
| [Test Coverage visualization](../../user/project/merge_requests/test_coverage_visualization.md) | See test coverage results for merge requests, in the file diff. |
| [Fail fast testing](../../user/project/merge_requests/fail_fast_testing.md#fail-fast-testing) | Run a subset of your RSpec test suite, so failed tests stop the pipeline before the full suite of tests run, saving resources. |
| Feature | Description |
|-------------------------------------------------------------------------|-------------|
| [Accessibility Testing](accessibility_testing.md) | Automatically report A11y violations for changed pages in merge requests. |
| [Browser Performance Testing](browser_performance_testing.md) | Quickly determine the browser performance impact of pending code changes. |
| [Load Performance Testing](load_performance_testing.md) | Quickly determine the server performance impact of pending code changes. |
| [Code Quality](code_quality.md) | Analyze your source code quality using the [Code Climate](https://codeclimate.com/) analyzer and show the Code Climate report right in the merge request widget area. |
| [Display arbitrary job artifacts](../yaml/index.md#artifactsexpose_as) | Configure CI pipelines with the `artifacts:expose_as` parameter to directly link to selected [artifacts](../pipelines/job_artifacts.md) in merge requests. |
| [Unit test reports](unit_test_reports.md) | Configure your CI jobs to use Unit test reports, and let GitLab display a report on the merge request so that it's easier and faster to identify the failure without having to check the entire job log. |
| [License Compliance](../../user/compliance/license_compliance/index.md) | Manage the licenses of your dependencies. |
| [Metrics Reports](metrics_reports.md) | Display the Metrics Report on the merge request so that it's fast and easier to identify changes to important metrics. |
| [Test Coverage visualization](test_coverage_visualization.md) | See test coverage results for merge requests, in the file diff. |
| [Fail fast testing](fail_fast_testing.md) | Run a subset of your RSpec test suite, so failed tests stop the pipeline before the full suite of tests run, saving resources. |
## Security Reports **(ULTIMATE)**
......
doc/ci/testing/load_performance_testing.md 0 → 100644
Просмотр файла @ 36084610
---
stage: Verify
group: Pipeline Insights
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
---
# Load Performance Testing **(PREMIUM)**
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/10683) in GitLab 13.2.
With Load Performance Testing, you can test the impact of any pending code changes
to your application's backend in [GitLab CI/CD](../index.md).
GitLab uses [k6](https://k6.io/), a free and open source
tool, for measuring the system performance of applications under
load.
Unlike [Browser Performance Testing](browser_performance_testing.md), which is
used to measure how web sites perform in client browsers, Load Performance Testing
can be used to perform various types of [load tests](https://k6.io/docs/#use-cases)
against application endpoints such as APIs, Web Controllers, and so on.
This can be used to test how the backend or the server performs at scale.
For example, you can use Load Performance Testing to perform many concurrent
GET calls to a popular API endpoint in your application to see how it performs.
## How Load Performance Testing works
First, define a job in your `.gitlab-ci.yml` file that generates the
[Load Performance report artifact](../yaml/artifacts_reports.md#artifactsreportsload_performance).
GitLab checks this report, compares key load performance metrics
between the source and target branches, and then shows the information in a merge request widget:
![Load Performance Widget](img/load_performance_testing.png)
Next, you need to configure the test environment and write the k6 test.
The key performance metrics that the merge request widget shows after the test completes are:
- Checks: The percentage pass rate of the [checks](https://k6.io/docs/using-k6/checks) configured in the k6 test.
- TTFB P90: The 90th percentile of how long it took to start receiving responses, aka the [Time to First Byte](https://en.wikipedia.org/wiki/Time_to_first_byte) (TTFB).
- TTFB P95: The 95th percentile for TTFB.
- RPS: The average requests per second (RPS) rate the test was able to achieve.
NOTE:
If the Load Performance report has no data to compare, such as when you add the
Load Performance job in your `.gitlab-ci.yml` for the very first time,
the Load Performance report widget doesn't display. It must have run at least
once on the target branch (`main`, for example), before it displays in a
merge request targeting that branch.
## Configure the Load Performance Testing job
Configuring your Load Performance Testing job can be broken down into several distinct parts:
- Determine the test parameters such as throughput, and so on.
- Set up the target test environment for load performance testing.
- Design and write the k6 test.
### Determine the test parameters
The first thing you need to do is determine the [type of load test](https://k6.io/docs/test-types/introduction)
you want to run, and how it will run (for example, the number of users, throughput, and so on).
Refer to the [k6 docs](https://k6.io/docs/), especially the [k6 testing guides](https://k6.io/docs/testing-guides),
for guidance on the above and more.
### Test Environment setup
A large part of the effort around load performance testing is to prepare the target test environment
for high loads. You should ensure it's able to handle the
[throughput](https://k6.io/blog/monthly-visits-concurrent-users) it will be tested with.
It's also typically required to have representative test data in the target environment
for the load performance test to use.
We strongly recommend [not running these tests against a production environment](https://k6.io/our-beliefs#load-test-in-a-pre-production-environment).
### Write the load performance test
After the environment is prepared, you can write the k6 test itself. k6 is a flexible
tool and can be used to run [many kinds of performance tests](https://k6.io/docs/test-types/introduction).
Refer to the [k6 documentation](https://k6.io/docs/) for detailed information on how to write tests.
### Configure the test in GitLab CI/CD
When your k6 test is ready, the next step is to configure the load performance
testing job in GitLab CI/CD. The easiest way to do this is to use the
[`Verify/Load-Performance-Testing.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Verify/Load-Performance-Testing.gitlab-ci.yml)
template that is included with GitLab.
NOTE:
For large scale k6 tests you need to ensure the GitLab Runner instance performing the actual
test is able to handle running the test. Refer to [k6's guidance](https://k6.io/docs/testing-guides/running-large-tests#hardware-considerations)
for spec details. The [default shared GitLab.com runners](../runners/saas/linux_saas_runner.md)
likely have insufficient specs to handle most large k6 tests.
This template runs the
[k6 Docker container](https://hub.docker.com/r/loadimpact/k6/) in the job and provides several ways to customize the
job.
An example configuration workflow:
1. Set up GitLab Runner to run Docker containers, like the
[Docker-in-Docker workflow](../docker/using_docker_build.md#use-docker-in-docker).
1. Configure the default Load Performance Testing CI/CD job in your `.gitlab-ci.yml` file.
You need to include the template and configure it with CI/CD variables:
```yaml
include:
template: Verify/Load-Performance-Testing.gitlab-ci.yml
load_performance:
variables:
K6_TEST_FILE: <PATH TO K6 TEST FILE IN PROJECT>
```
The above example creates a `load_performance` job in your CI/CD pipeline that runs
the k6 test.
NOTE:
For Kubernetes setups a different template should be used: [`Jobs/Load-Performance-Testing.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Load-Performance-Testing.gitlab-ci.yml).
k6 has [various options](https://k6.io/docs/using-k6/options) to configure how it will run tests, such as what throughput (RPS) to run with,
how long the test should run, and so on. Almost all options can be configured in the test itself, but as
you can also pass command line options via the `K6_OPTIONS` variable.
For example, you can override the duration of the test with a CLI option:
```yaml
include:
template: Verify/Load-Performance-Testing.gitlab-ci.yml
load_performance:
variables:
K6_TEST_FILE: <PATH TO K6 TEST FILE IN PROJECT>
K6_OPTIONS: '--duration 30s'
```
GitLab only displays the key performance metrics in the MR widget if k6's results are saved
via [summary export](https://k6.io/docs/results-visualization/json#summary-export)
as a [Load Performance report artifact](../yaml/artifacts_reports.md#artifactsreportsload_performance).
The latest Load Performance artifact available is always used, using the
summary values from the test.
If [GitLab Pages](../../user/project/pages/index.md) is enabled, you can view the report directly in your browser.
### Load Performance testing in Review Apps
The CI/CD YAML configuration example above works for testing against static environments,
but it can be extended to work with [review apps](../review_apps/index.md) or
[dynamic environments](../environments/index.md) with a few extra steps.
The best approach is to capture the dynamic URL in a [`.env` file](https://docs.docker.com/compose/env-file/)
as a job artifact to be shared, then use a custom CI/CD variable we've provided named `K6_DOCKER_OPTIONS`
to configure the k6 Docker container to use the file. With this, k6 can then use any
environment variables from the `.env` file in scripts using standard JavaScript,
such as: ``http.get(`${__ENV.ENVIRONMENT_URL}`)``.
For example:
1. In the `review` job:
1. Capture the dynamic URL and save it into a `.env` file, for example, `echo "ENVIRONMENT_URL=$CI_ENVIRONMENT_URL" >> review.env`.
1. Set the `.env` file to be a [job artifact](../pipelines/job_artifacts.md).
1. In the `load_performance` job:
1. Set it to depend on the review job, so it inherits the environment file.
1. Set the `K6_DOCKER_OPTIONS` variable with the [Docker CLI option for environment files](https://docs.docker.com/engine/reference/commandline/run/#set-environment-variables--e---env---env-file), for example `--env-file review.env`.
1. Configure the k6 test script to use the environment variable in it's steps.
Your `.gitlab-ci.yml` file might be similar to:
```yaml
stages:
- deploy
- performance
include:
template: Verify/Load-Performance-Testing.gitlab-ci.yml
review:
stage: deploy
environment:
name: review/$CI_COMMIT_REF_SLUG
url: http://$CI_ENVIRONMENT_SLUG.example.com
script:
- run_deploy_script
- echo "ENVIRONMENT_URL=$CI_ENVIRONMENT_URL" >> review.env
artifacts:
paths:
- review.env
rules:
- if: $CI_COMMIT_BRANCH # Modify to match your pipeline rules, or use `only/except` if needed.
load_performance:
dependencies:
- review
variables:
K6_DOCKER_OPTIONS: '--env-file review.env'
rules:
- if: $CI_COMMIT_BRANCH # Modify to match your pipeline rules, or use `only/except` if needed.
```
doc/ci/testing/metrics_reports.md 0 → 100644
Просмотр файла @ 36084610
---
stage: Verify
group: Pipeline Insights
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
---
# Metrics Reports **(PREMIUM)**
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9788) in GitLab 11.10. Requires GitLab Runner 11.10 and above.
GitLab provides a lot of great reporting tools for things like [merge requests](../../user/project/merge_requests/index.md) - [Unit test reports](unit_test_reports.md), [code quality](code_quality.md), and performance tests. While JUnit is a great open framework for tests that "pass" or "fail", it is also important to see other types of metrics from a given change.
You can configure your job to use custom Metrics Reports, and GitLab displays a report on the merge request so that it's easier and faster to identify changes without having to check the entire log.
![Metrics Reports](img/metrics_reports_v13_0.png)
## Use cases
Consider the following examples of data that can use Metrics Reports:
1. Memory usage
1. Load testing results
1. Code complexity
1. Code coverage stats
## How it works
Metrics for a branch are read from the latest metrics report artifact (default filename: `metrics.txt`) as string values.
For an MR, the values of these metrics from the feature branch are compared to the values from the target branch. Then they are displayed in the MR widget in this order:
- Existing metrics with changed values.
- Metrics that have been added by the MR. Marked with a **New** badge.
- Metrics that have been removed by the MR. Marked with a **Removed** badge.
- Existing metrics with unchanged values.
## How to set it up
Add a job that creates a [metrics report](../yaml/artifacts_reports.md#artifactsreportsmetrics) (default filename: `metrics.txt`). The file should conform to the [OpenMetrics](https://openmetrics.io/) format.
For example:
```yaml
metrics:
script:
- echo 'metric_name metric_value' > metrics.txt
artifacts:
reports:
metrics: metrics.txt
```
## Advanced Example
An advanced example of an OpenMetrics text file (from the [Prometheus documentation](https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exposition_formats.md#text-format-example))
renders in the merge request widget as:
![Metrics Reports Advanced](img/metrics_reports_advanced_v13_0.png)
## Troubleshooting
### Metrics reports did not change
You can see `Metrics reports did not change` when trying to view metrics reports in merge requests. Reasons for this are:
- The target branch for the merge request doesn't have a baseline metrics report for comparison.
- You don't have GitLab license at the Premium tier or above.
There is [an issue open](https://gitlab.com/gitlab-org/gitlab/-/issues/343065) to improve this message.
doc/ci/testing/test_coverage_visualization.md 0 → 100644
Просмотр файла @ 36084610
---
stage: Verify
group: Pipeline Insights
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
---
# Test coverage visualization **(FREE)**
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/3708) in GitLab 12.9.
> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/249811) in GitLab 13.5.
With the help of [GitLab CI/CD](../index.md), you can collect the test
coverage information of your favorite testing or coverage-analysis tool, and visualize
this information inside the file diff view of your merge requests (MRs). This will allow you
to see which lines are covered by tests, and which lines still require coverage, before the
MR is merged.
![Test Coverage Visualization Diff View](img/test_coverage_visualization_v12_9.png)
## How test coverage visualization works
Collecting the coverage information is done via GitLab CI/CD's
[artifacts reports feature](../yaml/index.md#artifactsreports).
You can specify one or more coverage reports to collect, including wildcard paths.
GitLab then takes the coverage information in all the files and combines it
together. Coverage files are parsed in a background job so there can be a delay
between pipeline completion and the visualization loading on the page.
For the coverage analysis to work, you have to provide a properly formatted
[Cobertura XML](https://cobertura.github.io/cobertura/) report to
[`artifacts:reports:coverage_report`](../yaml/artifacts_reports.md#artifactsreportscoverage_report).
This format was originally developed for Java, but most coverage analysis frameworks
for other languages have plugins to add support for it, like:
- [simplecov-cobertura](https://rubygems.org/gems/simplecov-cobertura) (Ruby)
- [gocover-cobertura](https://github.com/boumenot/gocover-cobertura) (Golang)
Other coverage analysis frameworks support the format out of the box, for example:
- [Istanbul](https://istanbul.js.org/docs/advanced/alternative-reporters/#cobertura) (JavaScript)
- [Coverage.py](https://coverage.readthedocs.io/en/coverage-5.0.4/cmd.html#xml-reporting) (Python)
- [PHPUnit](https://github.com/sebastianbergmann/phpunit-documentation-english/blob/master/src/textui.rst#command-line-options) (PHP)
Once configured, if you create a merge request that triggers a pipeline which collects
coverage reports, the coverage is shown in the diff view. This includes reports
from any job in any stage in the pipeline. The coverage displays for each line:
- `covered` (green): lines which have been checked at least once by tests
- `no test coverage` (orange): lines which are loaded but never executed
- no coverage information: lines which are non-instrumented or not loaded
Hovering over the coverage bar provides further information, such as the number
of times the line was checked by tests.
Uploading a test coverage report does not enable:
- [Test coverage results in merge requests](../pipelines/settings.md#merge-request-test-coverage-results).
- [Code coverage history](../pipelines/settings.md#view-code-coverage-history).
You must configure these separately.
### Limits
A limit of 100 `<source>` nodes for Cobertura format XML files applies. If your Cobertura report exceeds
100 nodes, there can be mismatches or no matches in the merge request diff view.
A single Cobertura XML file can be no more than 10MiB. For large projects, split the Cobertura XML into
smaller files. See [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/328772) for more details.
When submitting many files, it can take a few minutes for coverage to show on a merge request.
The visualization only displays after the pipeline is complete. If the pipeline has
a [blocking manual job](../jobs/job_control.md#types-of-manual-jobs), the
pipeline waits for the manual job before continuing and is not considered complete.
The visualization cannot be displayed if the blocking manual job did not run.
### Artifact expiration
By default, the [pipeline artifact](../pipelines/pipeline_artifacts.md#storage) used
to draw the visualization on the merge request expires **one week** after creation.
### Coverage report from child pipeline
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/363301) in GitLab 15.1 [with a flag](../../administration/feature_flags.md). Disabled by default.
FLAG:
On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../../administration/feature_flags.md) named `ci_child_pipeline_coverage_reports`.
On GitLab.com, this feature is not available.
The feature is not ready for production use.
If the test coverage is created in jobs that are in a child pipeline, the parent pipeline must use
`strategy: depend`.
```yaml
child_test_pipeline:
trigger:
include:
- local: path/to/child_pipeline.yml
- template: Security/SAST.gitlab-ci.yml
strategy: depend
```
### Automatic class path correction
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217664) in GitLab 13.8.
> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/284822) in GitLab 13.9.
The coverage report properly matches changed files only if the `filename` of a `class` element
contains the full path relative to the project root. However, in some coverage analysis frameworks,
the generated Cobertura XML has the `filename` path relative to the class package directory instead.
To make an intelligent guess on the project root relative `class` path, the Cobertura XML parser
attempts to build the full path by:
- Extracting a portion of the `source` paths from the `sources` element and combining them with the
class `filename` path.
- Checking if the candidate path exists in the project.
- Using the first candidate that matches as the class full path.
#### Path correction example
As an example, a project with:
- A full path of `test-org/test-project`.
- The following files relative to the project root:
```shell
Auth/User.cs
Lib/Utils/User.cs
src/main/java
```
In the:
- Cobertura XML, the `filename` attribute in the `class` element assumes the value is a relative
path to the project's root:
```xml
<class name="packet.name" filename="src/main/java" line-rate="0.0" branch-rate="0.0" complexity="5">
```
- `sources` from Cobertura XML, the following paths in the format
`<CI_BUILDS_DIR>/<PROJECT_FULL_PATH>/...`:
```xml
<sources>
<source>/builds/test-org/test-project/Auth</source>
<source>/builds/test-org/test-project/Lib/Utils</source>
</sources>
```
The parser:
- Extracts `Auth` and `Lib/Utils` from the `sources` and uses these to determine the `class` path
relative to the project root.
- Combines these extracted `sources` and the class filename. For example, if there is a `class`
element with the `filename` value of `User.cs`, the parser takes the first candidate path that
matches, which is `Auth/User.cs`.
- For each `class` element, attempts to look for a match for each extracted `source` path up to
100 iterations. If it reaches this limit without finding a matching path in the file tree, the
class is not included in the final coverage report.
NOTE:
Automatic class path correction only works on `source` paths in the format `<CI_BUILDS_DIR>/<PROJECT_FULL_PATH>/...`.
The `source` is ignored if the path does not follow this pattern. The parser assumes that the
`filename` of a `class` element contains the full path relative to the project root.
## Example test coverage configurations
This section provides test coverage configuration examples for different programming languages. You can also see a working example in
the [`coverage-report`](https://gitlab.com/gitlab-org/ci-sample-projects/coverage-report/) demonstration project.
### JavaScript example
The following [`.gitlab-ci.yml`](../yaml/index.md) example uses [Mocha](https://mochajs.org/)
JavaScript testing and [nyc](https://github.com/istanbuljs/nyc) coverage-tooling to
generate the coverage artifact:
```yaml
test:
script:
- npm install
- npx nyc --reporter cobertura mocha
artifacts:
reports:
coverage_report:
coverage_format: cobertura
path: coverage/cobertura-coverage.xml
```
### Java and Kotlin examples
#### Maven example
The following [`.gitlab-ci.yml`](../yaml/index.md) example for Java or Kotlin uses [Maven](https://maven.apache.org/)
to build the project and [JaCoCo](https://www.eclemma.org/jacoco/) coverage-tooling to
generate the coverage artifact.
You can check the [Docker image configuration and scripts](https://gitlab.com/haynes/jacoco2cobertura) if you want to build your own image.
GitLab expects the artifact in the Cobertura format, so you have to execute a few
scripts before uploading it. The `test-jdk11` job tests the code and generates an
XML artifact. The `coverage-jdk-11` job converts the artifact into a Cobertura report:
```yaml
test-jdk11:
stage: test
image: maven:3.6.3-jdk-11
script:
- mvn $MAVEN_CLI_OPTS clean org.jacoco:jacoco-maven-plugin:prepare-agent test jacoco:report
artifacts:
paths:
- target/site/jacoco/jacoco.xml
coverage-jdk11:
# Must be in a stage later than test-jdk11's stage.
# The `visualize` stage does not exist by default.
# Please define it first, or choose an existing stage like `deploy`.
stage: visualize
image: registry.gitlab.com/haynes/jacoco2cobertura:1.0.7
script:
# convert report from jacoco to cobertura, using relative project path
- python /opt/cover2cover.py target/site/jacoco/jacoco.xml $CI_PROJECT_DIR/src/main/java/ > target/site/cobertura.xml
needs: ["test-jdk11"]
artifacts:
reports:
coverage_report:
coverage_format: cobertura
path: target/site/cobertura.xml
```
#### Gradle example
The following [`.gitlab-ci.yml`](../yaml/index.md) example for Java or Kotlin uses [Gradle](https://gradle.org/)
to build the project and [JaCoCo](https://www.eclemma.org/jacoco/) coverage-tooling to
generate the coverage artifact.
You can check the [Docker image configuration and scripts](https://gitlab.com/haynes/jacoco2cobertura) if you want to build your own image.
GitLab expects the artifact in the Cobertura format, so you have to execute a few
scripts before uploading it. The `test-jdk11` job tests the code and generates an
XML artifact. The `coverage-jdk-11` job converts the artifact into a Cobertura report:
```yaml
test-jdk11:
stage: test
image: gradle:6.6.1-jdk11
script:
- 'gradle test jacocoTestReport' # jacoco must be configured to create an xml report
artifacts:
paths:
- build/jacoco/jacoco.xml
coverage-jdk11:
# Must be in a stage later than test-jdk11's stage.
# The `visualize` stage does not exist by default.
# Please define it first, or chose an existing stage like `deploy`.
stage: visualize
image: registry.gitlab.com/haynes/jacoco2cobertura:1.0.7
script:
# convert report from jacoco to cobertura, using relative project path
- python /opt/cover2cover.py build/jacoco/jacoco.xml $CI_PROJECT_DIR/src/main/java/ > build/cobertura.xml
needs: ["test-jdk11"]
artifacts:
reports:
coverage_report:
coverage_format: cobertura
path: build/cobertura.xml
```
### Python example
The following [`.gitlab-ci.yml`](../yaml/index.md) example for Python uses [pytest-cov](https://pytest-cov.readthedocs.io/) to collect test coverage data and [coverage.py](https://coverage.readthedocs.io/) to convert the report to use full relative paths.
The information isn't displayed without the conversion.
This example assumes that the code for your package is in `src/` and your tests are in `tests.py`:
```yaml
run tests:
stage: test
image: python:3
script:
- pip install pytest pytest-cov
- coverage run -m pytest
- coverage report
- coverage xml
coverage: '/(?i)total.*? (100(?:\.0+)?\%|[1-9]?\d(?:\.\d+)?\%)$/'
artifacts:
reports:
coverage_report:
coverage_format: cobertura
path: coverage.xml
```
### PHP example
The following [`.gitlab-ci.yml`](../yaml/index.md) example for PHP uses [PHPUnit](https://phpunit.readthedocs.io/)
to collect test coverage data and generate the report.
With a minimal [`phpunit.xml`](https://phpunit.readthedocs.io/en/9.5/configuration.html) file (you may reference
[this example repository](https://gitlab.com/yookoala/code-coverage-visualization-with-php/)), you can run the test and
generate the `coverage.xml`:
```yaml
run tests:
stage: test
image: php:latest
variables:
XDEBUG_MODE: coverage
before_script:
- apt-get update && apt-get -yq install git unzip zip libzip-dev zlib1g-dev
- docker-php-ext-install zip
- pecl install xdebug && docker-php-ext-enable xdebug
- php -r "copy('https://getcomposer.org/installer', 'composer-setup.php');"
- php composer-setup.php --install-dir=/usr/local/bin --filename=composer
- composer install
- composer require --dev phpunit/phpunit phpunit/php-code-coverage
script:
- php ./vendor/bin/phpunit --coverage-text --coverage-cobertura=coverage.cobertura.xml
artifacts:
reports:
coverage_report:
coverage_format: cobertura
path: coverage.cobertura.xml
```
[Codeception](https://codeception.com/), through PHPUnit, also supports generating Cobertura report with
[`run`](https://codeception.com/docs/reference/Commands#run). The path for the generated file
depends on the `--coverage-cobertura` option and [`paths`](https://codeception.com/docs/reference/Configuration#paths)
configuration for the [unit test suite](https://codeception.com/docs/05-UnitTests). Configure `.gitlab-ci.yml`
to find Cobertura in the appropriate path.
### C/C++ example
The following [`.gitlab-ci.yml`](../yaml/index.md) example for C/C++ with
`gcc` or `g++` as the compiler uses [`gcovr`](https://gcovr.com/en/stable/) to generate the coverage
output file in Cobertura XML format.
This example assumes:
- That the `Makefile` is created by `cmake` in the `build` directory,
within another job in a previous stage.
(If you use `automake` to generate the `Makefile`,
then you need to call `make check` instead of `make test`.)
- `cmake` (or `automake`) has set the compiler option `--coverage`.
```yaml
run tests:
stage: test
script:
- cd build
- make test
- gcovr --xml-pretty --exclude-unreachable-branches --print-summary -o coverage.xml --root ${CI_PROJECT_DIR}
coverage: /^\s*lines:\s*\d+.\d+\%/
artifacts:
name: ${CI_JOB_NAME}-${CI_COMMIT_REF_NAME}-${CI_COMMIT_SHA}
expire_in: 2 days
reports:
coverage_report:
coverage_format: cobertura
path: build/coverage.xml
```
### Go example
The following [`.gitlab-ci.yml`](../yaml/index.md) example for Go uses:
- [`go test`](https://go.dev/doc/tutorial/add-a-test) to run tests.
- [`gocover-cobertura`](https://github.com/boumenot/gocover-cobertura) to convert Go's coverage profile into the Cobertura XML format.
This example assumes that [Go modules](https://go.dev/ref/mod)
are being used. Please note that the `-covermode count` option does not work with the `-race` flag.
If you want to generate code coverage while also using the `-race` flag, you must switch to
`-covermode atomic` which is slower than `-covermode count`. See [this blog post](https://go.dev/blog/cover)
for more details.
```yaml
run tests:
stage: test
image: golang:1.17
script:
- go install
- go test ./... -coverprofile=coverage.txt -covermode count
- go get github.com/boumenot/gocover-cobertura
- go run github.com/boumenot/gocover-cobertura < coverage.txt > coverage.xml
artifacts:
reports:
coverage_report:
coverage_format: cobertura
path: coverage.xml
```
### Ruby example
The following [`.gitlab-ci.yml`](../yaml/index.md) example for Ruby uses
- [`rspec`](https://rspec.info/) to run tests.
- [`simplecov`](https://github.com/simplecov-ruby/simplecov) and [`simplecov-cobertura`](https://github.com/dashingrocket/simplecov-cobertura)
to record the coverage profile and create a report in the Cobertura XML format.
This example assumes:
- That [`bundler`](https://bundler.io/) is being used for dependency management.
The `rspec`, `simplecov` and `simplecov-cobertura` gems have been added to your `Gemfile`.
- The `CoberturaFormatter` has been added to your `SimpleCov.formatters`
configuration within the `spec_helper.rb` file.
```yaml
run tests:
stage: test
image: ruby:3.1
script:
- bundle install
- bundle exec rspec
artifacts:
reports:
coverage_report:
coverage_format: cobertura
path: coverage/coverage.xml
```
## Troubleshooting
### Test coverage visualization not displayed
If the test coverage visualization is not displayed in the diff view, you can check
the coverage report itself and verify that:
- The file you are viewing in the diff view is mentioned in the coverage report.
- The `source` and `filename` nodes in the report follows the [expected structure](#automatic-class-path-correction)
to match the files in your repository.
Report artifacts are not downloadable by default. If you want the report to be downloadable
from the job details page, add your coverage report to the artifact `paths`:
```yaml
artifacts:
paths:
- coverage/cobertura-coverage.xml
reports:
coverage_report:
coverage_format: cobertura
path: coverage/cobertura-coverage.xml
```
doc/ci/yaml/artifacts_reports.md
Просмотр файла @ 36084610
......@@ -100,7 +100,7 @@ instead.
Use `coverage_report` to collect coverage report in Cobertura format.
The `cobertura` report collects [Cobertura coverage XML files](../../user/project/merge_requests/test_coverage_visualization.md).
The `cobertura` report collects [Cobertura coverage XML files](../testing/test_coverage_visualization.md).
Cobertura was originally developed for Java, but there are many third-party ports for other languages such as
JavaScript, Python, and Ruby.
......@@ -116,7 +116,7 @@ artifacts:
The collected coverage report is uploaded to GitLab as an artifact.
GitLab can display the results of coverage report in the merge request
[diff annotations](../../user/project/merge_requests/test_coverage_visualization.md).
[diff annotations](../testing/test_coverage_visualization.md).
## `artifacts:reports:codequality`
......@@ -131,7 +131,7 @@ GitLab can display the results of:
- Only one report in:
- The merge request [diff annotations](../testing/code_quality.md#code-quality-in-diff-view).
Track progress on adding support for multiple reports in [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/328257).
- The [full report](../metrics_reports.md). Track progress on adding support for multiple reports in
- The [full report](../testing/metrics_reports.md). Track progress on adding support for multiple reports in
[this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/9014).
## `artifacts:reports:container_scanning` **(ULTIMATE)**
......@@ -263,21 +263,21 @@ GitLab can display the results of one or more reports in:
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/35260) in GitLab 13.2.
> - Requires GitLab Runner 11.5 and above.
The `load_performance` report collects [Load Performance Testing metrics](../../user/project/merge_requests/load_performance_testing.md).
The `load_performance` report collects [Load Performance Testing metrics](../testing/load_performance_testing.md).
The report is uploaded to GitLab as an artifact.
GitLab can display the results of only one report in the merge request
[load testing widget](../../user/project/merge_requests/load_performance_testing.md#how-load-performance-testing-works).
[load testing widget](../testing/load_performance_testing.md#how-load-performance-testing-works).
GitLab cannot display the combined results of multiple `load_performance` reports.
## `artifacts:reports:metrics` **(PREMIUM)**
The `metrics` report collects [Metrics](../metrics_reports.md). The collected Metrics report uploads to GitLab as an
The `metrics` report collects [Metrics](../testing/metrics_reports.md). The collected Metrics report uploads to GitLab as an
artifact.
GitLab can display the results of one or more reports in the merge request
[metrics reports widget](../../ci/metrics_reports.md#metrics-reports).
[metrics reports widget](../testing/metrics_reports.md#metrics-reports).
## `artifacts:reports:requirements` **(ULTIMATE)**
......
doc/development/performance.md
Просмотр файла @ 36084610
......@@ -29,7 +29,7 @@ consistent performance of GitLab. Refer to the [Index](#performance-documentatio
- [Browser performance testing guidelines](../ci/testing/browser_performance_testing.md)
- [`gdk measure` and `gdk measure-workflow`](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/gdk_commands.md#measure-performance)
- QA:
- [Load performance testing](../user/project/merge_requests/load_performance_testing.md)
- [Load performance testing](../ci/testing/load_performance_testing.md)
- [GitLab Performance Tool project](https://gitlab.com/gitlab-org/quality/performance)
- [Review apps performance metrics](../development/testing_guide/review_apps.md#performance-metrics)
- Monitoring & Overview:
......@@ -581,7 +581,7 @@ called `memory-on-boot`. ([Read an example job.](https://gitlab.com/gitlab-org/g
You may find the results:
- On the merge request **Overview** tab, in the merge request reports area, in the
**Metrics Reports** [dropdown list](../ci/metrics_reports.md).
**Metrics Reports** [dropdown list](../ci/testing/metrics_reports.md).
- In the `memory-on-boot` artifacts for a full report and a dependency breakdown.
`derailed_benchmarks` also provides other methods to investigate memory. To learn more,
......
doc/topics/autodevops/stages.md
Просмотр файла @ 36084610
......@@ -357,7 +357,7 @@ Any browser performance differences between the source and target branches are a
> Introduced in GitLab 13.2.
Auto [Load Performance Testing](../../user/project/merge_requests/load_performance_testing.md)
Auto [Load Performance Testing](../../ci/testing/load_performance_testing.md)
measures the server performance of an application with the
[k6 container](https://hub.docker.com/r/loadimpact/k6/),
creates a JSON report including several key result metrics, and
......@@ -368,7 +368,7 @@ written that's tailored to your specific application. The test also needs to be
configured so it can pick up the environment's dynamic URL via a CI/CD variable.
Any load performance test result differences between the source and target branches are also
[shown in the merge request widget](../../user/project/merge_requests/load_performance_testing.md).
[shown in the merge request widget](../../user/project/merge_requests/widgets.md).
## Auto Deploy
......
doc/user/project/merge_requests/fail_fast_testing.md
Просмотр файла @ 36084610
---
stage: Verify
group: Pipeline Insights
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
redirect_to: '../../../ci/testing/fail_fast_testing.md'
remove_date: '2022-08-31'
---
# Fail Fast Testing **(PREMIUM)**
This document was moved to [another location](../../../ci/testing/fail_fast_testing.md).
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/198550) in GitLab 13.1.
For applications that use RSpec for running tests, we've introduced the `Verify/Failfast`
[template to run subsets of your test suite](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/templates/Verify/FailFast.gitlab-ci.yml),
based on the changes in your merge request.
The template uses the [test_file_finder (`tff`) gem](https://gitlab.com/gitlab-org/ci-cd/test_file_finder/)
that accepts a list of files as input, and returns a list of spec (test) files
that it believes to be relevant to the input files.
`tff` is designed for Ruby on Rails projects, so the `Verify/FailFast` template is
configured to run when changes to Ruby files are detected. By default, it runs in
the [`.pre` stage](../../../ci/yaml/index.md#stage-pre) of a GitLab CI/CD pipeline,
before all other stages.
## Example use case
Fail fast testing is useful when adding new functionality to a project and adding
new automated tests.
Your project could have hundreds of thousands of tests that take a long time to complete.
You may be confident that a new test will pass, but you have to wait for all the tests
to complete to verify it. This could take an hour or more, even when using parallelization.
Fail fast testing gives you a faster feedback loop from the pipeline. It lets you
know quickly that the new tests are passing and the new functionality did not break
other tests.
## Requirements
This template requires:
- A project built in Rails that uses RSpec for testing.
- CI/CD configured to:
- Use a Docker image with Ruby available.
- Use [Merge request pipelines](../../../ci/pipelines/merge_request_pipelines.md#prerequisites)
- [Merged results pipelines](../../../ci/pipelines/merged_results_pipelines.md#enable-merged-results-pipelines)
enabled in the project settings.
- A Docker image with Ruby available. The template uses `image: ruby:2.6` by default, but you [can override](../../../ci/yaml/includes.md#override-included-configuration-values) this.
## Configuring Fast RSpec Failure
We use the following plain RSpec configuration as a starting point. It installs all the
project gems and executes `rspec`, on merge request pipelines only.
```yaml
rspec-complete:
stage: test
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
script:
- bundle install
- bundle exec rspec
```
To run the most relevant specs first instead of the whole suite, [`include`](../../../ci/yaml/index.md#include)
the template by adding the following to your CI/CD configuration:
```yaml
include:
- template: Verify/FailFast.gitlab-ci.yml
```
To customize the job, specific options may be set to override the template. For example, to override the default Docker image:
```yaml
include:
- template: Verify/FailFast.gitlab-ci.yml
rspec-rails-modified-path-specs:
image: custom-docker-image-with-ruby
```
### Example test loads
For illustrative purposes, let's say our Rails app spec suite consists of 100 specs per model for ten models.
If no Ruby files are changed:
- `rspec-rails-modified-paths-specs` does not run any tests.
- `rspec-complete` runs the full suite of 1000 tests.
If one Ruby model is changed, for example `app/models/example.rb`, then `rspec-rails-modified-paths-specs`
runs the 100 tests for `example.rb`:
- If all of these 100 tests pass, then the full `rspec-complete` suite of 1000 tests is allowed to run.
- If any of these 100 tests fail, they fail quickly, and `rspec-complete` does not run any tests.
The final case saves resources and time as the full 1000 test suite does not run.
<!-- This redirect file can be deleted after <2022-09-22>. -->
<!-- Redirects that point to other docs in the same project expire in three months. -->
<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. -->
<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html -->
doc/user/project/merge_requests/load_performance_testing.md
Просмотр файла @ 36084610
---
stage: Verify
group: Pipeline Insights
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
redirect_to: '../../../ci/testing/load_performance_testing.md'
remove_date: '2022-08-31'
---
# Load Performance Testing **(PREMIUM)**
This document was moved to [another location](../../../ci/testing/load_performance_testing.md).
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/10683) in GitLab 13.2.
With Load Performance Testing, you can test the impact of any pending code changes
to your application's backend in [GitLab CI/CD](../../../ci/index.md).
GitLab uses [k6](https://k6.io/), a free and open source
tool, for measuring the system performance of applications under
load.
Unlike [Browser Performance Testing](../../../ci/testing/browser_performance_testing.md), which is
used to measure how web sites perform in client browsers, Load Performance Testing
can be used to perform various types of [load tests](https://k6.io/docs/#use-cases)
against application endpoints such as APIs, Web Controllers, and so on.
This can be used to test how the backend or the server performs at scale.
For example, you can use Load Performance Testing to perform many concurrent
GET calls to a popular API endpoint in your application to see how it performs.
## How Load Performance Testing works
First, define a job in your `.gitlab-ci.yml` file that generates the
[Load Performance report artifact](../../../ci/yaml/artifacts_reports.md#artifactsreportsload_performance).
GitLab checks this report, compares key load performance metrics
between the source and target branches, and then shows the information in a merge request widget:
![Load Performance Widget](img/load_performance_testing.png)
Next, you need to configure the test environment and write the k6 test.
The key performance metrics that the merge request widget shows after the test completes are:
- Checks: The percentage pass rate of the [checks](https://k6.io/docs/using-k6/checks) configured in the k6 test.
- TTFB P90: The 90th percentile of how long it took to start receiving responses, aka the [Time to First Byte](https://en.wikipedia.org/wiki/Time_to_first_byte) (TTFB).
- TTFB P95: The 95th percentile for TTFB.
- RPS: The average requests per second (RPS) rate the test was able to achieve.
NOTE:
If the Load Performance report has no data to compare, such as when you add the
Load Performance job in your `.gitlab-ci.yml` for the very first time,
the Load Performance report widget doesn't display. It must have run at least
once on the target branch (`main`, for example), before it displays in a
merge request targeting that branch.
## Configure the Load Performance Testing job
Configuring your Load Performance Testing job can be broken down into several distinct parts:
- Determine the test parameters such as throughput, and so on.
- Set up the target test environment for load performance testing.
- Design and write the k6 test.
### Determine the test parameters
The first thing you need to do is determine the [type of load test](https://k6.io/docs/test-types/introduction)
you want to run, and how it will run (for example, the number of users, throughput, and so on).
Refer to the [k6 docs](https://k6.io/docs/), especially the [k6 testing guides](https://k6.io/docs/testing-guides),
for guidance on the above and more.
### Test Environment setup
A large part of the effort around load performance testing is to prepare the target test environment
for high loads. You should ensure it's able to handle the
[throughput](https://k6.io/blog/monthly-visits-concurrent-users) it will be tested with.
It's also typically required to have representative test data in the target environment
for the load performance test to use.
We strongly recommend [not running these tests against a production environment](https://k6.io/our-beliefs#load-test-in-a-pre-production-environment).
### Write the load performance test
After the environment is prepared, you can write the k6 test itself. k6 is a flexible
tool and can be used to run [many kinds of performance tests](https://k6.io/docs/test-types/introduction).
Refer to the [k6 documentation](https://k6.io/docs/) for detailed information on how to write tests.
### Configure the test in GitLab CI/CD
When your k6 test is ready, the next step is to configure the load performance
testing job in GitLab CI/CD. The easiest way to do this is to use the
[`Verify/Load-Performance-Testing.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Verify/Load-Performance-Testing.gitlab-ci.yml)
template that is included with GitLab.
NOTE:
For large scale k6 tests you need to ensure the GitLab Runner instance performing the actual
test is able to handle running the test. Refer to [k6's guidance](https://k6.io/docs/testing-guides/running-large-tests#hardware-considerations)
for spec details. The [default shared GitLab.com runners](../../../ci/runners/saas/linux_saas_runner.md)
likely have insufficient specs to handle most large k6 tests.
This template runs the
[k6 Docker container](https://hub.docker.com/r/loadimpact/k6/) in the job and provides several ways to customize the
job.
An example configuration workflow:
1. Set up GitLab Runner to run Docker containers, like the
[Docker-in-Docker workflow](../../../ci/docker/using_docker_build.md#use-docker-in-docker).
1. Configure the default Load Performance Testing CI/CD job in your `.gitlab-ci.yml` file.
You need to include the template and configure it with CI/CD variables:
```yaml
include:
template: Verify/Load-Performance-Testing.gitlab-ci.yml
load_performance:
variables:
K6_TEST_FILE: <PATH TO K6 TEST FILE IN PROJECT>
```
The above example creates a `load_performance` job in your CI/CD pipeline that runs
the k6 test.
NOTE:
For Kubernetes setups a different template should be used: [`Jobs/Load-Performance-Testing.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Load-Performance-Testing.gitlab-ci.yml).
k6 has [various options](https://k6.io/docs/using-k6/options) to configure how it will run tests, such as what throughput (RPS) to run with,
how long the test should run, and so on. Almost all options can be configured in the test itself, but as
you can also pass command line options via the `K6_OPTIONS` variable.
For example, you can override the duration of the test with a CLI option:
```yaml
include:
template: Verify/Load-Performance-Testing.gitlab-ci.yml
load_performance:
variables:
K6_TEST_FILE: <PATH TO K6 TEST FILE IN PROJECT>
K6_OPTIONS: '--duration 30s'
```
GitLab only displays the key performance metrics in the MR widget if k6's results are saved
via [summary export](https://k6.io/docs/results-visualization/json#summary-export)
as a [Load Performance report artifact](../../../ci/yaml/artifacts_reports.md#artifactsreportsload_performance).
The latest Load Performance artifact available is always used, using the
summary values from the test.
If [GitLab Pages](../pages/index.md) is enabled, you can view the report directly in your browser.
### Load Performance testing in Review Apps
The CI/CD YAML configuration example above works for testing against static environments,
but it can be extended to work with [review apps](../../../ci/review_apps) or
[dynamic environments](../../../ci/environments) with a few extra steps.
The best approach is to capture the dynamic URL in a [`.env` file](https://docs.docker.com/compose/env-file/)
as a job artifact to be shared, then use a custom CI/CD variable we've provided named `K6_DOCKER_OPTIONS`
to configure the k6 Docker container to use the file. With this, k6 can then use any
environment variables from the `.env` file in scripts using standard JavaScript,
such as: ``http.get(`${__ENV.ENVIRONMENT_URL}`)``.
For example:
1. In the `review` job:
1. Capture the dynamic URL and save it into a `.env` file, for example, `echo "ENVIRONMENT_URL=$CI_ENVIRONMENT_URL" >> review.env`.
1. Set the `.env` file to be a [job artifact](../../../ci/pipelines/job_artifacts.md#job-artifacts).
1. In the `load_performance` job:
1. Set it to depend on the review job, so it inherits the environment file.
1. Set the `K6_DOCKER_OPTIONS` variable with the [Docker CLI option for environment files](https://docs.docker.com/engine/reference/commandline/run/#set-environment-variables--e---env---env-file), for example `--env-file review.env`.
1. Configure the k6 test script to use the environment variable in it's steps.
Your `.gitlab-ci.yml` file might be similar to:
```yaml
stages:
- deploy
- performance
include:
template: Verify/Load-Performance-Testing.gitlab-ci.yml
review:
stage: deploy
environment:
name: review/$CI_COMMIT_REF_SLUG
url: http://$CI_ENVIRONMENT_SLUG.example.com
script:
- run_deploy_script
- echo "ENVIRONMENT_URL=$CI_ENVIRONMENT_URL" >> review.env
artifacts:
paths:
- review.env
rules:
- if: $CI_COMMIT_BRANCH # Modify to match your pipeline rules, or use `only/except` if needed.
load_performance:
dependencies:
- review
variables:
K6_DOCKER_OPTIONS: '--env-file review.env'
rules:
- if: $CI_COMMIT_BRANCH # Modify to match your pipeline rules, or use `only/except` if needed.
```
<!-- This redirect file can be deleted after <2022-09-22>. -->
<!-- Redirects that point to other docs in the same project expire in three months. -->
<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. -->
<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html -->
doc/user/project/merge_requests/test_coverage_visualization.md
Просмотр файла @ 36084610
---
stage: Verify
group: Pipeline Insights
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
redirect_to: '../../../ci/testing/test_coverage_visualization.md'
remove_date: '2022-08-31'
---
# Test coverage visualization **(FREE)**
This document was moved to [another location](../../../ci/testing/test_coverage_visualization.md).
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/3708) in GitLab 12.9.
> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/249811) in GitLab 13.5.
With the help of [GitLab CI/CD](../../../ci/index.md), you can collect the test
coverage information of your favorite testing or coverage-analysis tool, and visualize
this information inside the file diff view of your merge requests (MRs). This will allow you
to see which lines are covered by tests, and which lines still require coverage, before the
MR is merged.
![Test Coverage Visualization Diff View](img/test_coverage_visualization_v12_9.png)
## How test coverage visualization works
Collecting the coverage information is done via GitLab CI/CD's
[artifacts reports feature](../../../ci/yaml/index.md#artifactsreports).
You can specify one or more coverage reports to collect, including wildcard paths.
GitLab then takes the coverage information in all the files and combines it
together. Coverage files are parsed in a background job so there can be a delay
between pipeline completion and the visualization loading on the page.
For the coverage analysis to work, you have to provide a properly formatted
[Cobertura XML](https://cobertura.github.io/cobertura/) report to
[`artifacts:reports:coverage_report`](../../../ci/yaml/artifacts_reports.md#artifactsreportscoverage_report).
This format was originally developed for Java, but most coverage analysis frameworks
for other languages have plugins to add support for it, like:
- [simplecov-cobertura](https://rubygems.org/gems/simplecov-cobertura) (Ruby)
- [gocover-cobertura](https://github.com/boumenot/gocover-cobertura) (Golang)
Other coverage analysis frameworks support the format out of the box, for example:
- [Istanbul](https://istanbul.js.org/docs/advanced/alternative-reporters/#cobertura) (JavaScript)
- [Coverage.py](https://coverage.readthedocs.io/en/coverage-5.0.4/cmd.html#xml-reporting) (Python)
- [PHPUnit](https://github.com/sebastianbergmann/phpunit-documentation-english/blob/master/src/textui.rst#command-line-options) (PHP)
Once configured, if you create a merge request that triggers a pipeline which collects
coverage reports, the coverage is shown in the diff view. This includes reports
from any job in any stage in the pipeline. The coverage displays for each line:
- `covered` (green): lines which have been checked at least once by tests
- `no test coverage` (orange): lines which are loaded but never executed
- no coverage information: lines which are non-instrumented or not loaded
Hovering over the coverage bar provides further information, such as the number
of times the line was checked by tests.
Uploading a test coverage report does not enable:
- [Test coverage results in merge requests](../../../ci/pipelines/settings.md#merge-request-test-coverage-results).
- [Code coverage history](../../../ci/pipelines/settings.md#view-code-coverage-history).
You must configure these separately.
### Limits
A limit of 100 `<source>` nodes for Cobertura format XML files applies. If your Cobertura report exceeds
100 nodes, there can be mismatches or no matches in the merge request diff view.
A single Cobertura XML file can be no more than 10MiB. For large projects, split the Cobertura XML into
smaller files. See [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/328772) for more details.
When submitting many files, it can take a few minutes for coverage to show on a merge request.
The visualization only displays after the pipeline is complete. If the pipeline has
a [blocking manual job](../../../ci/jobs/job_control.md#types-of-manual-jobs), the
pipeline waits for the manual job before continuing and is not considered complete.
The visualization cannot be displayed if the blocking manual job did not run.
### Artifact expiration
By default, the [pipeline artifact](../../../ci/pipelines/pipeline_artifacts.md#storage) used
to draw the visualization on the merge request expires **one week** after creation.
### Coverage report from child pipeline
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/363301) in GitLab 15.1 [with a flag](../../../administration/feature_flags.md). Disabled by default.
FLAG:
On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../../../administration/feature_flags.md) named `ci_child_pipeline_coverage_reports`.
On GitLab.com, this feature is not available.
The feature is not ready for production use.
If the test coverage is created in jobs that are in a child pipeline, the parent pipeline must use
`strategy: depend`.
```yaml
child_test_pipeline:
trigger:
include:
- local: path/to/child_pipeline.yml
- template: Security/SAST.gitlab-ci.yml
strategy: depend
```
### Automatic class path correction
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217664) in GitLab 13.8.
> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/284822) in GitLab 13.9.
The coverage report properly matches changed files only if the `filename` of a `class` element
contains the full path relative to the project root. However, in some coverage analysis frameworks,
the generated Cobertura XML has the `filename` path relative to the class package directory instead.
To make an intelligent guess on the project root relative `class` path, the Cobertura XML parser
attempts to build the full path by:
- Extracting a portion of the `source` paths from the `sources` element and combining them with the
class `filename` path.
- Checking if the candidate path exists in the project.
- Using the first candidate that matches as the class full path.
#### Path correction example
As an example, a project with:
- A full path of `test-org/test-project`.
- The following files relative to the project root:
```shell
Auth/User.cs
Lib/Utils/User.cs
src/main/java
```
In the:
- Cobertura XML, the `filename` attribute in the `class` element assumes the value is a relative
path to the project's root:
```xml
<class name="packet.name" filename="src/main/java" line-rate="0.0" branch-rate="0.0" complexity="5">
```
- `sources` from Cobertura XML, the following paths in the format
`<CI_BUILDS_DIR>/<PROJECT_FULL_PATH>/...`:
```xml
<sources>
<source>/builds/test-org/test-project/Auth</source>
<source>/builds/test-org/test-project/Lib/Utils</source>
</sources>
```
The parser:
- Extracts `Auth` and `Lib/Utils` from the `sources` and uses these to determine the `class` path
relative to the project root.
- Combines these extracted `sources` and the class filename. For example, if there is a `class`
element with the `filename` value of `User.cs`, the parser takes the first candidate path that
matches, which is `Auth/User.cs`.
- For each `class` element, attempts to look for a match for each extracted `source` path up to
100 iterations. If it reaches this limit without finding a matching path in the file tree, the
class is not included in the final coverage report.
NOTE:
Automatic class path correction only works on `source` paths in the format `<CI_BUILDS_DIR>/<PROJECT_FULL_PATH>/...`.
The `source` is ignored if the path does not follow this pattern. The parser assumes that the
`filename` of a `class` element contains the full path relative to the project root.
## Example test coverage configurations
This section provides test coverage configuration examples for different programming languages. You can also see a working example in
the [`coverage-report`](https://gitlab.com/gitlab-org/ci-sample-projects/coverage-report/) demonstration project.
### JavaScript example
The following [`.gitlab-ci.yml`](../../../ci/yaml/index.md) example uses [Mocha](https://mochajs.org/)
JavaScript testing and [nyc](https://github.com/istanbuljs/nyc) coverage-tooling to
generate the coverage artifact:
```yaml
test:
script:
- npm install
- npx nyc --reporter cobertura mocha
artifacts:
reports:
coverage_report:
coverage_format: cobertura
path: coverage/cobertura-coverage.xml
```
### Java and Kotlin examples
#### Maven example
The following [`.gitlab-ci.yml`](../../../ci/yaml/index.md) example for Java or Kotlin uses [Maven](https://maven.apache.org/)
to build the project and [JaCoCo](https://www.eclemma.org/jacoco/) coverage-tooling to
generate the coverage artifact.
You can check the [Docker image configuration and scripts](https://gitlab.com/haynes/jacoco2cobertura) if you want to build your own image.
GitLab expects the artifact in the Cobertura format, so you have to execute a few
scripts before uploading it. The `test-jdk11` job tests the code and generates an
XML artifact. The `coverage-jdk-11` job converts the artifact into a Cobertura report:
```yaml
test-jdk11:
stage: test
image: maven:3.6.3-jdk-11
script:
- mvn $MAVEN_CLI_OPTS clean org.jacoco:jacoco-maven-plugin:prepare-agent test jacoco:report
artifacts:
paths:
- target/site/jacoco/jacoco.xml
coverage-jdk11:
# Must be in a stage later than test-jdk11's stage.
# The `visualize` stage does not exist by default.
# Please define it first, or choose an existing stage like `deploy`.
stage: visualize
image: registry.gitlab.com/haynes/jacoco2cobertura:1.0.7
script:
# convert report from jacoco to cobertura, using relative project path
- python /opt/cover2cover.py target/site/jacoco/jacoco.xml $CI_PROJECT_DIR/src/main/java/ > target/site/cobertura.xml
needs: ["test-jdk11"]
artifacts:
reports:
coverage_report:
coverage_format: cobertura
path: target/site/cobertura.xml
```
#### Gradle example
The following [`.gitlab-ci.yml`](../../../ci/yaml/index.md) example for Java or Kotlin uses [Gradle](https://gradle.org/)
to build the project and [JaCoCo](https://www.eclemma.org/jacoco/) coverage-tooling to
generate the coverage artifact.
You can check the [Docker image configuration and scripts](https://gitlab.com/haynes/jacoco2cobertura) if you want to build your own image.
GitLab expects the artifact in the Cobertura format, so you have to execute a few
scripts before uploading it. The `test-jdk11` job tests the code and generates an
XML artifact. The `coverage-jdk-11` job converts the artifact into a Cobertura report:
```yaml
test-jdk11:
stage: test
image: gradle:6.6.1-jdk11
script:
- 'gradle test jacocoTestReport' # jacoco must be configured to create an xml report
artifacts:
paths:
- build/jacoco/jacoco.xml
coverage-jdk11:
# Must be in a stage later than test-jdk11's stage.
# The `visualize` stage does not exist by default.
# Please define it first, or chose an existing stage like `deploy`.
stage: visualize
image: registry.gitlab.com/haynes/jacoco2cobertura:1.0.7
script:
# convert report from jacoco to cobertura, using relative project path
- python /opt/cover2cover.py build/jacoco/jacoco.xml $CI_PROJECT_DIR/src/main/java/ > build/cobertura.xml
needs: ["test-jdk11"]
artifacts:
reports:
coverage_report:
coverage_format: cobertura
path: build/cobertura.xml
```
### Python example
The following [`.gitlab-ci.yml`](../../../ci/yaml/index.md) example for Python uses [pytest-cov](https://pytest-cov.readthedocs.io/) to collect test coverage data and [coverage.py](https://coverage.readthedocs.io/) to convert the report to use full relative paths.
The information isn't displayed without the conversion.
This example assumes that the code for your package is in `src/` and your tests are in `tests.py`:
```yaml
run tests:
stage: test
image: python:3
script:
- pip install pytest pytest-cov
- coverage run -m pytest
- coverage report
- coverage xml
coverage: '/(?i)total.*? (100(?:\.0+)?\%|[1-9]?\d(?:\.\d+)?\%)$/'
artifacts:
reports:
coverage_report:
coverage_format: cobertura
path: coverage.xml
```
### PHP example
The following [`.gitlab-ci.yml`](../../../ci/yaml/index.md) example for PHP uses [PHPUnit](https://phpunit.readthedocs.io/)
to collect test coverage data and generate the report.
With a minimal [`phpunit.xml`](https://phpunit.readthedocs.io/en/9.5/configuration.html) file (you may reference
[this example repository](https://gitlab.com/yookoala/code-coverage-visualization-with-php/)), you can run the test and
generate the `coverage.xml`:
```yaml
run tests:
stage: test
image: php:latest
variables:
XDEBUG_MODE: coverage
before_script:
- apt-get update && apt-get -yq install git unzip zip libzip-dev zlib1g-dev
- docker-php-ext-install zip
- pecl install xdebug && docker-php-ext-enable xdebug
- php -r "copy('https://getcomposer.org/installer', 'composer-setup.php');"
- php composer-setup.php --install-dir=/usr/local/bin --filename=composer
- composer install
- composer require --dev phpunit/phpunit phpunit/php-code-coverage
script:
- php ./vendor/bin/phpunit --coverage-text --coverage-cobertura=coverage.cobertura.xml
artifacts:
reports:
coverage_report:
coverage_format: cobertura
path: coverage.cobertura.xml
```
[Codeception](https://codeception.com/), through PHPUnit, also supports generating Cobertura report with
[`run`](https://codeception.com/docs/reference/Commands#run). The path for the generated file
depends on the `--coverage-cobertura` option and [`paths`](https://codeception.com/docs/reference/Configuration#paths)
configuration for the [unit test suite](https://codeception.com/docs/05-UnitTests). Configure `.gitlab-ci.yml`
to find Cobertura in the appropriate path.
### C/C++ example
The following [`.gitlab-ci.yml`](../../../ci/yaml/index.md) example for C/C++ with
`gcc` or `g++` as the compiler uses [`gcovr`](https://gcovr.com/en/stable/) to generate the coverage
output file in Cobertura XML format.
This example assumes:
- That the `Makefile` is created by `cmake` in the `build` directory,
within another job in a previous stage.
(If you use `automake` to generate the `Makefile`,
then you need to call `make check` instead of `make test`.)
- `cmake` (or `automake`) has set the compiler option `--coverage`.
```yaml
run tests:
stage: test
script:
- cd build
- make test
- gcovr --xml-pretty --exclude-unreachable-branches --print-summary -o coverage.xml --root ${CI_PROJECT_DIR}
coverage: /^\s*lines:\s*\d+.\d+\%/
artifacts:
name: ${CI_JOB_NAME}-${CI_COMMIT_REF_NAME}-${CI_COMMIT_SHA}
expire_in: 2 days
reports:
coverage_report:
coverage_format: cobertura
path: build/coverage.xml
```
### Go example
The following [`.gitlab-ci.yml`](../../../ci/yaml/index.md) example for Go uses:
- [`go test`](https://go.dev/doc/tutorial/add-a-test) to run tests.
- [`gocover-cobertura`](https://github.com/boumenot/gocover-cobertura) to convert Go's coverage profile into the Cobertura XML format.
This example assumes that [Go modules](https://go.dev/ref/mod)
are being used. Please note that the `-covermode count` option does not work with the `-race` flag.
If you want to generate code coverage while also using the `-race` flag, you must switch to
`-covermode atomic` which is slower than `-covermode count`. See [this blog post](https://go.dev/blog/cover)
for more details.
```yaml
run tests:
stage: test
image: golang:1.17
script:
- go install
- go test ./... -coverprofile=coverage.txt -covermode count
- go get github.com/boumenot/gocover-cobertura
- go run github.com/boumenot/gocover-cobertura < coverage.txt > coverage.xml
artifacts:
reports:
coverage_report:
coverage_format: cobertura
path: coverage.xml
```
### Ruby example
The following [`.gitlab-ci.yml`](../../../ci/yaml/index.md) example for Ruby uses
- [`rspec`](https://rspec.info/) to run tests.
- [`simplecov`](https://github.com/simplecov-ruby/simplecov) and [`simplecov-cobertura`](https://github.com/dashingrocket/simplecov-cobertura)
to record the coverage profile and create a report in the Cobertura XML format.
This example assumes:
- That [`bundler`](https://bundler.io/) is being used for dependency management.
The `rspec`, `simplecov` and `simplecov-cobertura` gems have been added to your `Gemfile`.
- The `CoberturaFormatter` has been added to your `SimpleCov.formatters`
configuration within the `spec_helper.rb` file.
```yaml
run tests:
stage: test
image: ruby:3.1
script:
- bundle install
- bundle exec rspec
artifacts:
reports:
coverage_report:
coverage_format: cobertura
path: coverage/coverage.xml
```
## Troubleshooting
### Test coverage visualization not displayed
If the test coverage visualization is not displayed in the diff view, you can check
the coverage report itself and verify that:
- The file you are viewing in the diff view is mentioned in the coverage report.
- The `source` and `filename` nodes in the report follows the [expected structure](#automatic-class-path-correction)
to match the files in your repository.
Report artifacts are not downloadable by default. If you want the report to be downloadable
from the job details page, add your coverage report to the artifact `paths`:
```yaml
artifacts:
paths:
- coverage/cobertura-coverage.xml
reports:
coverage_report:
coverage_format: cobertura
path: coverage/cobertura-coverage.xml
```
<!-- This redirect file can be deleted after <2022-09-22>. -->
<!-- Redirects that point to other docs in the same project expire in three months. -->
<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. -->
<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html -->
Поддерживает Markdown
0% или .
You are about to add 0 people to the discussion. Proceed with caution.
Сначала завершите редактирование этого сообщения!
Пожалуйста, зарегистрируйтесь или чтобы прокомментировать