Xray Test Management
eazyBI for Jira
Xray for Jira is a complete, full-featured Atlassian verified test management add-on for Jira. Xray supports both manual and automated tests and a complete testing life cycle: test planning, test designing, and test execution to help developers improve the quality of their systems.
Xray for Jira is an add-on developed by Xblend, an Atlassian Platinum Marketplace Partner.
On this page:
Xray Data Import
Data Import in Server version
If you have installed Xray add-on for Jira Server then in Jira import options, section Add-ons, you will see an options Xray. Select the checkbox to import Xray data and after the next data import, Xray specific measures and dimensions will be available for your reports. You can also check or uncheck some of Xray specific custom fields for data import.
Data Import in Cloud version
eazyBI uses API keys to get data from Xray Cloud.
- In Jira Cloud, go to section Apps → Xray -> API Keys and create API key. For the API key select user with Jira administrator rights.
- In eazyBI, Jira import options section Add-ons you will see an option Xray which is available when you have installed Xray Test Management app for Jira. Enter generated Client Id and Client secret.
After the next data import, Xray specific measures and dimensions will be available for your reports.
Data import in Private eazyBI.
You can import Xray custom fields also when using Private eazyBI. Add the following settings to the configuration file config/eazybi.toml to enable Xray data import. There are more details on the needed configuration for data from Jira apps.
Limit imported test runs
To reduce import time and the testing data in the cube, you can import only the most recent test runs.
Set a relative time limit for test runs like 3 months to import test runs from the last three months or a particular date like Jun 14 2020 to set a date range from that date until now. Test runs are imported if they are created (test is added to the test execution) within the time limit (for scheduled test executions) or with started on date within the time limit (for completed or in progress test runs). If there is no limit set, then all test runs are imported.
You can set a limit to all eazyBI accounts through advanced settings.
[jira.xray] test_executions_limit = "3 months"
Another option is setting the limit for an individual eazyBI account. In the import options, add the period in the field "Imported Test Run limit".
If import limits are applied for the whole eazyBI instance using advanced settings, then the limit in import options won't exceed the limit in advanced settings. In such a way, the overall limits can be set on the eazyBI account, and, in an individual account, this time can be reduced more.
Dimensions
After the import the following Xray specific dimension will be added:
Xray Requirement UPDATED | Requirement issues with issue types selected in Xray configuration "Requirement Issue Types"; for example, story, feature, etc. The dimension is organized into Project and Requirement levels. Use this dimension with the measure Xray Requirements created count to see all covered and not covered requirements. |
|---|---|
| Xray Test Plan | Test plan issues that have assigned at least one test or executions with tests. The dimension is organized by default hierarchy (in Project and Plan levels) or by Test Plan boar hierarchy (in Project, Plan, Folder, and Subfolder). |
| Xray Test | Tests issues; all created issues with Xray-specific issue type "Test". The dimension is organized by default hierarchy (in Project and Test levels) or by Test repository hierarchy (in Project, Folder, Subfolder, and Test). |
| Xray Test Set | Test set issues that have at least one test. This dimension is organized into Project and Set levels. |
| Xray Test Pre-condition | Test pre-condition issues. This dimension is organized into Project and Pre-condition levels. |
| Xray Test Execution | Test execution issues that have assigned at least one test. This dimension is organized into Project and Execution levels. |
Xray Defect | Defect issues that are found during testing and linked through a test execution screen. This dimension is organized in Project and Defect levels. The Xray Defect dimension works well with execution-related measures, for example, Xray Test Run, Xray defect count. |
Xray Defect Priority NEW | Current defect priority of defects that are found during testing and linked through a test execution screen. This dimension is organized in Project and Defect levels. The Xray Defect Priority dimension works well with execution-related measures, for example, Xray Test Run, Xray defect count. Xray Defect Priority dimension available on Cloud and starting from the eazyBI version 6.6. |
Xray Defect Status NEW | Current defect statuses of defects that are found during testing and linked through a test execution screen. This dimension has a secondary hierarchy that groups statuses by Category. The Xray Defect Status dimension works well with execution-related measures, for example, Xray Test Run, Xray defect count. Xray Defect Status dimension available on Cloud and starting from the eazyBI version 6.6. |
Xray Test Run Assignee | Users who are assigned to test runs. |
| Xray Test Executed by | Users who executed tests. |
Xray Test Run Status | Test execution statuses – by default, there are TODO, PASS, EXECUTING, FAIL, and ABORTED statuses, but also there can be other user-defined statuses. The Xray Test Run Status dimension works well with execution-related measures, for example, Xray Test Run. The former dimension name in old versions was Xray Test Execution Status. |
Requirement Status NEW | Testing status of requirement issue (feature, story, etc.) based on Xray calculated custom field "Requirement status". If you change the configuration for calculating requirement status in Jira, ensure to re-import data in eazyBI to get and refresh all updated values. The Requirement Status dimension is available since eazyBI version 6.6. Dimension is not available for the Jira Cloud version. |
TestRunStatus NEW | Testing status of Test issue based on Xray calculated custom field "Requirement status". The TestRunStatus dimension is available since eazyBI version 6.6. Dimension is not available for the Jira Cloud version. |
| Xray Pre-condition Type | Types of pre-conditions - Manual, Cucumber. |
| Xray Test Type | Types of tests (Manual, Cucumber, Generic, etc.). The dimension is organized into Type and Scenario Type levels. |
Xray Test Execution Test Environment Test Environments | Test environment where tests are executed or scheduled for execution. The dimension name on Cloud is Xray Test Execution Test Environment. The dimension name on the Data Center and Server match the issue custom field name Test Environment. |
| Revision | System revision for test execution. Dimension is not available for the Jira Cloud version. |
Custom Dimensions
If you do not find Xray custom field Test Environments in import options, you may import Xray custom field Test Environments using Advanced settings for custom fields. Use the following custom field definition example (replacing NNNNN with a corresponding field ID in Jira):
[jira.customfield_NNNNN]
data_type = "string"
dimension = true
multiple_values = true
name = "Xray Revision"
dimension_annotations = {group = "Xray"}
Measures
The following Xray specific measures will be imported. These measures can be analyzed by the mentioned Xray specific dimensions as well as by Issue dimension (where individual tests executions are registered as issues with "Test Execution" type and tests as issues with "Test" type) and by other standard dimension members which correspond to the issue. Note that, measures used in the report defines the context of standard dimensions. For example:
- When using execution related measures, like, Xray Tests executed, Xray Test Runs, Xray Tests defect count, then standard dimension Status is applied to Xray Test Execution as execution is the core to gather information about executed tests, execution outcomes, fix versions, found defects, executors, etc.
- When using test related measures, like, Xray Tests created, then standard dimension Status is related to Xray Test.
- When using requirement related measures, like, Xray Requirements created, then standard dimension Status is related to the requirement issue (story, bugs, or others you have marked as requirements).
| Xray Tests created | Count of issues with type "Test". On the Time dimension, data are grouped by the issue creation date of the test. Used with Jira standard dimensions, like Project, Label, Status, Priority, etc., groups data by test issue fields. |
|---|---|
| Xray Test Executions created | Count of issues with type "Test Execution". On the Time dimension, data are grouped by the test execution issue creation date. Used with Jira standard dimensions, like Project, Label, Status, Priority, etc., groups data by test execution issue fields. |
Xray Requirements created NEW | Count of requirement issues that match issue types selected in Xray configuration "Requirement Issue Types". On the Time dimension, data are grouped by the issue creation date of the requirement issue. Used with Jira standard dimensions, like Project, Label, Status, Priority, etc., groups data by requirement issue fields. Measure Xray requirements created available on Cloud and starting eazyBI version 6.6. |
| Xray Tests defect count | How many defects are registered for test execution (including individual test steps). A unique count of different defects is returned for each test run and aggregated at higher levels. A defect should be linked through the Xray test execution screen. On the Time dimension, data are grouped by the test run date. |
| Xray Tests scheduled | How many tests are scheduled within selected dimensions but not yet executed (test has no dates for execution "Started" and "Finished"). If a test is scheduled for two separate test executions, then eazyBI count it as one test. On the Time dimension, data are grouped by the date when test is added to the execution. |
| Xray Tests executing | How many tests are executing within selected dimensions (test has execution "Started" date and no "Finished" date). If a test is executing in two separate test executions, then eazyBI count it as one test. On the Time dimension, data are grouped by the date when test execution (run) is started. |
| Xray Tests executed | How many tests are executed within selected dimensions (test has both execution dates "Started" and "Finished"). If a tests is executed in several test executions, then eazyBI count it as one test. On the Time dimension, data are grouped by the date when test execution (run) is finished. |
| Xray Tests with executions | How many tests have at least one test execution (scheduled, in progress or completed) within selected dimensions. If a tests has several test executions, then eazyBI count it as one test. On the Time dimension data are grouped by:
|
| Xray Test Runs | How many times have tests been executed, are executing, or are scheduled for execution. On the Time dimension data are grouped by:
|
| Xray Tests executed first date | Date and time for the first executed test within selected dimensions.
|
| Xray Tests executed last date | Date and time for the last executed test within selected dimensions.
|
| Xray average Test Run duration | How much time (hours and minutes) it took to execute a test on average. |
| Xray Tests execution summary | Test execution statuses within selected dimensions. If a test has several test executions with different results, then eazyBI counts it as one for each resulting status. |
| Xray Total Test Run duration | Hidden measures that could be used in the calculated measures. Total duration (hours and minutes) it took to run tests. |
Properties
Also following properties will be created.
| Xray Test last executed date | The last execution date for a test issue. It is issue property and value does not change according to selected Test Execution, Test Plan or other dimension members. Used together with Xray Test dimension. |
|---|---|
| Xray Test last executed by | The last executor for a test issue. It is issue property and value does not change according to selected Test Execution, Test Plan or other dimension members. Used together with Xray Test dimension. |
| Xray Test last execution status | The last execution status for a test issue. It is issue property and value does not change according to selected Test Execution, Test Plan or other dimension members. Used together with Xray Test dimension. |
| Xray Test Execution Begin Date | Begin date of a test execution issue. This is an execution property and should be used together with Xray Test Execution dimension. Property is not available for Jira Cloud version. You can import Begin Date as issue property "Xray Begin date" and access it through the calculated measure. |
| Xray Test Execution End Date | End date of a test execution issue. This is an execution property and should be used together with Xray Test Execution dimension. Property is not available for Jira Cloud version. You can import End Date as issue property "Xray End Date" and access it through the calculated measure. |
| Xray Test Execution Revision | Current revision assigned to a tests execution issue. This is an execution property and should be used together with Xray Test Execution dimension. Property is not available for Jira Cloud version. You can import field as issue property "Xray Revision" and access it through the calculated measure. |
| Xray Test Execution Test Environments | Current test environment assigned to a tests execution issue. This is an execution property and should be used together with Xray Test Execution dimension. Property is not available for Jira Cloud version. |
| Xray Execution Test Plans | Key of test plan to which a test execution issue is assigned. This is an execution property and should be used together with Xray Test Execution dimension. |
Xray Test Test Plans | List of test plans in which test is executed or scheduled for execution. This is a test property and should be used together with Xray Test dimension. |
| Xray Test Test Type | Current test type of a test issue. This is a test property and should be used together with Xray Test dimension. Property is not available for Jira Cloud version. |
| Xray Test Cucumber Test Type | Current Cucumber tests type of a test issue if it has a test type "Cucumber". This is a test property and should be used together with Xray Test dimension. Property is not available for Jira Cloud version. |
Xray Test Plan Begin Date | Begin date of a test plan issue. This is a test plan property and should be used together with Xray Test Plan dimension. Property is not available for Jira Cloud version. You can import Begin Date as issue property "Xray Begin Date" and access it through the calculated measure. |
| Xray Test Plan End Date | End date of a tests plan issue. This is a test plan property and should be used together with Xray Test Plan dimension. Property is not available for Jira Cloud version. You can import End Date as issue property "Xray End Date" and access it through the calculated measure. |
| Xray Defect status | Current status of a defect issue. This is a defect property and should be used together with the Xray Defect dimension. |
Xray Defect priority NEW | Current priority of a defect issue. This is a defect property and should be used together with the Xray Defect dimension. The Xray Defect priority is available on the Jira Cloud and since eazyBI version 6.6 (soon to be released). |
| Xray Defect created | Creation date and time of a defect issue. This is a defect property and should be used together with the Xray Defect dimension. |
| Xray Defect assignee | Current status of a defect issue. This is a defect property and should be used together with the Xray Defect dimension. |
Xray Requirement Testing Status NEW | Testing status of requirement issue based on Xray calculated custom field "Requirement Status". This is a requirement issue property and should be used together with Xray Requriement dimension. The Xray Requirement Testing Status is available since eazyBI version 6.6. Property is not available for the Jira Cloud version. |
Xray Test Execution Overall Execution Status NEW | The aggregated execution status for the tests in the test execution based on Xray calculated custom field "Test Execution Status" (or "Overall Execution Status"). This is a test execution property and should be used together with the Xray Test Execution dimension. The Xray Test Execution Overall Execution Status is available since eazyBI version 6.6. Property is not available for the Jira Cloud version. |
Xray Test Plan Overall Execution Status NEW | The aggregated execution status for the tests in the test plan based on Xray calculated custom field "Test Plan Status" (or "Overall Execution Status"). This is a test plan property and should be used together with the Xray Test Plan dimension. The Xray Test Plan Overall Execution Status is available since eazyBI version 6.6. Property is not available for the Jira Cloud version. |
Xray Test Set Overall Execution Status NEW | The aggregated execution status for the tests in the test set based on Xray calculated custom field "Test Set Status" (or "Overall Execution Status"). This is a test plan property and should be used together with the Xray Test Set dimension. The Xray Test Set Overall Execution Status is available since eazyBI version 6.6. Property is not available for the Jira Cloud version. |
Issue properties
Xray tests, test executions, as well as defects, requirements, and test plans are Jira issues and are also represented in the Issue dimension. You can define calculated measure to get any issue property that is imported in eazyBI also for Xray-specific dimensions. To access issue property, use function get().
For example, to get priority for Xray Requirement, copy a formula of issue property Issue Priority and replace dimension name [Issue] with [Xray Requirement] like this:
[Priority].[Priority].getMemberNameByKey(
[Xray Requirement].CurrentHierarchyMember.get('Priority ID')
)
Another example, to get Begin Date date for Xray Test Plan, copy a formula of the issue property Issue Xray Begin Date and replace dimension name [Issue] with [Xray Test Plan] like this:
[Xray Test Plan].CurrentHierarchyMember.get('Xray Begin Date')
Sample reports
If you selected the Import sample reports option, then additional sample reports are created in the Samples Xray folder. Please try these reports to see which measures would be useful for your specific needs. If you would like to customize some reports, then save them and change the name (as the default sample reports are overwritten during each Jira import).
See also detailed description on how to build non-standard reports:
- Xray Requirement coverage report
- Xray Test burnup for Test Plan report