From: stark, steven Date: Mon, 7 Jan 2019 23:14:51 +0000 (-0800) Subject: [VVP] ADD how to contribute information X-Git-Tag: 5.0.0~88 X-Git-Url: https://gerrit.onap.org/r/gitweb?p=vvp%2Fvalidation-scripts.git;a=commitdiff_plain;h=b07019fe0808b6a40930a6116c184ec0f1a0d885 [VVP] ADD how to contribute information Change-Id: I7a2dbd7f110d3c129521eb59d53b19c30cc3b8a2 Issue-ID: VVP-125 Signed-off-by: stark, steven --- diff --git a/README.rst b/README.rst index 8c3e782..87348ea 100644 --- a/README.rst +++ b/README.rst @@ -5,6 +5,8 @@ Manual Heat Template Validation =============================== +.. contents:: + validation-scripts ------------------ @@ -49,7 +51,7 @@ To validate Heat templates just run this the command under the folder ``ice_vali where ```` is the full path to a folder containing heat templates. Output -______ +###### After performing a validation, an output folder will be created. @@ -60,18 +62,18 @@ of the ONAP VNF Heat Template Guideline violations. If there are no violations, the report will say ``No validation errors found.`` Interpreting the Output -_______________________ +####################### The report file will have 4 columns for details about a violation, and one row for each violation. Below contains details about each column. File -#### +~~~~ This is the file(s) that contained the violation Error Message -############# +~~~~~~~~~~~~~ This shows the test and brief error message from the validation script that contained the violation. There is a ``Full Details`` button to show the @@ -80,7 +82,7 @@ about what element is involved with the violation (such as the parameter name, resource id, etc...). Requirement(s) -############## +~~~~~~~~~~~~~~ This column contains the requirement(s) that each test/violation is mapped to. These requirements are taken directly from the VNF Requirements @@ -88,7 +90,7 @@ project Heat Orchestration Template Guidelines section. Resolution Steps -################ +~~~~~~~~~~~~~~~~ For some violations, there are pre-defined resolution steps that indicate what action the user should take to resolve the violation. @@ -97,18 +99,210 @@ indicate what action the user should take to resolve the violation. can be resolved simply by reviewing the requirements that have been violated in the previous column. +How to Contribute +_________________ + +Before getting started +###################### + +Objective +~~~~~~~~~ + +**The objective for the VVP test suite is for each +test to directly correlate with at least one requirement in the** +`VNF Requirements `__ +**project in ONAP. If the test you intend to write doesn't +have a corresponding requirement in the VNF Requirements project, consider +making a contribution to that project first.** + +Convenience vs Convention +~~~~~~~~~~~~~~~~~~~~~~~~~ + +There are a lot of ways to write tests. Priorities for the VVP test suite are + + - Accuracy + - User Comprehension + +The test suite is often used by people who don't write code, or people +who aren't devoted to writing python validation tests. + +The output of failed validation tests can be difficult to read, so +keep that in mind when you are deciding whether to create another +level of abstraction vs having some code duplication or verbose tests. + +Developer Guide +############### + +File Name +~~~~~~~~~ + +Test files are written in python, and should go into the +``/validation-scripts/ice_validator/tests/`` directory. They should be prefixed +with ``test_``. If not, ``pytest`` will not discover your +test if you don't follow this convention. + +Test Name +~~~~~~~~~ + +Tests are functions defined in the test file, and also must be prefixed with +``test_``. If not, ``pytest`` will not collect them during execution. +For example: + +**test_my_new_requirement_file.py** + +.. code-block:: python + + def test_my_new_requirement(): + +Requirement Decorator +~~~~~~~~~~~~~~~~~~~~~ + +Each test function should be decorated with a requirement ID from the +VNF Requirements project. The following is required to be imported at +the top of the test file: + +``from .helpers import validates`` + +Then, your test function should be decorated like this: + +.. code-block:: python + + @validates("R-123456", + "R-123457") # these requirement IDs should come from the VNFRQTS project + def test_my_new_requirement(): + +This decorator is used at the end of the test suite execution to generate a +report that includes the requirements that were violated. If a test is not +decorated it is unclear what the reason for a failure is, and the +implication is that the test is not needed. + +Test Parameters +~~~~~~~~~~~~~~~ + +Each test should be parameterized based on what artifact is being validated. +Available parameters are enumerated in +``/validation-scripts/ice_validator/tests/parameterizers.py``. Below is a description +of the most commonly used: + + - ``heat_template``: parameter is the full path name for a file with the + extenstion ``.yaml`` or ``.yml``, + if the file also has a corresponding file with the same name but + extension ``.env``. + - ``yaml_file``: parameter is the full path name for a file with the + extenstion ``.yaml`` or ``.yml`` + - ``yaml_files``: parameter is a list of all files with the extenstion + ``.yaml`` or ``.yml``. + - ``volume_template``: parameter is the full path name for a file name + that ends with ``_volume`` and the extension ``.yaml`` or ``.yml``. + +There are many others that can also be used, check ``parameterizers.py`` for +the full list. + +The parameter that you decide to use determines how many times a test is +executed, and what data is available to validate. For example, if the +test suite is executed against a directory with 10 ``.yaml`` files, and +a test is using the parameter ``yaml_file``, the test will be executed +once for each file, for a total of 10 executions. If the parameter +``yaml_files`` (note the plural) is used instead, the test will +only execute once. + +Here's an example for how to parameterize a test: + +.. code-block:: python + + @validates("R-123456", + "R-123457") + def test_my_new_requirement(yaml_file): # this test will execute for each .yaml or .yml + +Collecting Failures +~~~~~~~~~~~~~~~~~~~ + +To raise a violation to ``pytest`` to be collected and included on the final +violation report, use the ``assert`` statement. Example: + +.. code-block:: python + + @validates("R-123456", + "R-123457") + def test_my_new_requirement(yaml_file): + my test logic + ... + ... + ... + + assert not failure_condition, error_message + +As one of the VVP priorities is User Comprehension, the ``error_message`` +should be readable and include helpful information for triaging the failure, +such as the ``yaml_file``, the parameter the test was checking, etc... + +If the assert statement fails, the failure is collected by ``pytest``, and the +associated requirements and error_message are included in the final report. + +Optional: Pytest Markers and Validation Profiles +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The VVP test suite has the concept of a ``base`` test. These are used as +sanity tests and are executed before the other tests, and if they fail the +test suite execution is halted. If you are writing a ``base`` test, mark your +test like this: + +.. code-block:: python + + import pytest + + @pytest.mark.base # this is the base test marker + @validates("R-123456") + def test_my_new_requirement(): + +The VVP test suite also has the concept of a ``validation profile`` to +define what set of tests to execute. The way it works is by using ``pytest`` +markers. + +By default, all ``base`` tests and non-marked tests are executed. If you want +an additional profile to run, pass the command line argument: + +``--validation-profile=`` + +This will execute all ``base`` tests, non-marked tests, +and tests marked like the following: + +.. code-block:: python + + import pytest + + @pytest.mark. # this is an additional pytest marker + @validates("R-123456") + def test_my_new_requirement(): + +This should be used sparingly, and in practice consider reviewing a requirement +with the VNF Requirements team before adding a test to a validation profile. Self-Test Suite -_______________ +~~~~~~~~~~~~~~~ + +The VVP test suite includes an extensive self-test suite. This can be +executed by running + +``$ pytest --self-test tests/`` + +This self test suite is used as a check for any new or modified tests. -The ``ice_validator`` includes an extensive self-test suite. It is a -**requirement** for any additions or changes to the test suite to -successfully and cleanly complete a tox run. Simply run ``tox`` from -the project root as: +If you are adding a new test, a new self-test ``fixture`` **MUST** be created +in the directory ``/validation-scripts/ice_validator/tests/fixtures``. The +directory should be named identical to the new python file (without the ``.py`` +extension), and it should contain 2 subdirectories: ``pass`` and ``fail``. -``$ tox`` +These directories should include heat templates that pass and fail the new test. +For Example, if I have created a new test called ``test_my_new_requirement.py`` +, I should create: -You can also run it under the folder ``ice_validator``: +``/validation-scripts/ice_validator/tests/fixtures/test_my_new_requirement/pass`` +``/validation-scripts/ice_validator/tests/fixtures/test_my_new_requirement/pass/pass.yaml`` +``/validation-scripts/ice_validator/tests/fixtures/test_my_new_requirement/fail`` +``/validation-scripts/ice_validator/tests/fixtures/test_my_new_requirement/fail/fail.yaml`` -``$ pytest --self-test`` +When executing the self-test suite, the templates in these folders are +expected to pass and fail, respectively, **ONLY** for the corresponding test. +They don't need to pass the whole test suite.