From ac35c043b29a6c65a0c9ce932aa12e1f406ff44c Mon Sep 17 00:00:00 2001 From: "stark, steven" Date: Wed, 17 Apr 2019 09:25:06 -0700 Subject: [PATCH] [VVP] updating VVP Readme to point to RTD Change-Id: Ic2a696058f7acce88ddd881a132bb4ab8a277e56 Issue-ID: VVP-196 Signed-off-by: stark, steven --- README.rst | 304 ++----------------------------------------------------------- 1 file changed, 6 insertions(+), 298 deletions(-) diff --git a/README.rst b/README.rst index 397285d..a3309ad 100644 --- a/README.rst +++ b/README.rst @@ -2,307 +2,15 @@ .. http://creativecommons.org/licenses/by/4.0 .. Copyright 2019 AT&T Intellectual Property. All rights reserved. -Manual Heat Template Validation -=============================== - -.. contents:: - -validation-scripts ------------------- +VVP Validation Scripts +====================== This project contains validation scripts to test that a set of Heat Templates adheres to -the ONAP VNF Heat Orchestration Template guidelines. +the ONAP VNF Heat Orchestration Template Guidelines and Requirements. For more information on the ONAP Heat Orchestration -Template Guidelines, vist the `Heat Guidelines `__ - -About -_____ - -The validation scripts project allows performing heat template -validation without installing the full VVP platform. The following -instructions apply to running these validation scripts in that manner. - - -Installation -____________ - -This software is not platform dependent and can be run in a Windows, Unix or -OS X environment. - -Satisfy Dependencies -#################### - - - These can be installed using pip (assuming pip is installed) with the command: - -``$ pip install -r requirements.txt`` - -Use -___ - -Clone this project. - -To validate Heat templates just run this the command under the folder ``ice_validator``: - -``$ pytest --tap-stream --template-directory=`` - -where ```` is the full path to a folder containing heat templates. - -Output -###### - -After performing a validation, an output folder will be created. - -``/path/to/validation-scripts/ice_validator/output/`` - -This folder will contain a file ``report.html`` which contains a list of all -of the ONAP VNF Heat Template Guideline violations. If there are no violations, -the report will say ``No validation errors found.`` - -Interpreting the Output -####################### - -The report file will have 4 columns for details about a violation, and one -row for each violation. Below contains details about each column. - -File -~~~~ - -This is the file(s) that contained the violation - -Error Message -~~~~~~~~~~~~~ - -This shows the test and brief error message from the validation script that -contained the violation. There is a ``Full Details`` button to show the -complete raw test output. The error message will also contain details -about what element is involved with the violation (such as the parameter -name, resource id, etc...). - -Requirement(s) -~~~~~~~~~~~~~~ - -This column contains the requirement(s) that each test/violation is -mapped to. These requirements are taken directly from the VNF Requirements -project Heat Orchestration Template Guidelines section. - - -Resolution Steps -~~~~~~~~~~~~~~~~ - -For some violations, there are pre-defined resolution steps that -indicate what action the user should take to resolve the violation. - -**Note**: Not all violations will have resolution steps. Most violations -can be resolved simply by reviewing the requirements that have been violated -in the previous column. - -How to Contribute -_________________ - -Before getting started -###################### - -Objective -~~~~~~~~~ - -**The objective for the VVP test suite is for each -test to directly correlate with at least one requirement in the** -`VNF Requirements `__ -**project in ONAP. If the test you intend to write doesn't -have a corresponding requirement in the VNF Requirements project, consider -making a contribution to that project first.** - -Convenience vs Convention -~~~~~~~~~~~~~~~~~~~~~~~~~ - -There are a lot of ways to write tests. Priorities for the VVP test suite are - - - Accuracy - - User Comprehension - -The test suite is often used by people who don't write code, or people -who aren't devoted to writing python validation tests. - -The output of failed validation tests can be difficult to read, so -keep that in mind when you are deciding whether to create another -level of abstraction vs having some code duplication or verbose tests. - -Developer Guide -############### - -File Name -~~~~~~~~~ - -Test files are written in python, and should go into the -``/validation-scripts/ice_validator/tests/`` directory. They should be prefixed -with ``test_``. If not, ``pytest`` will not discover your -test if you don't follow this convention. - -Test Name -~~~~~~~~~ - -Tests are functions defined in the test file, and also must be prefixed with -``test_``. If not, ``pytest`` will not collect them during execution. -For example: - -**test_my_new_requirement_file.py** - -.. code-block:: python - - def test_my_new_requirement(): - -Requirement Decorator -~~~~~~~~~~~~~~~~~~~~~ - -Each test function should be decorated with a requirement ID from the -VNF Requirements project. The following is required to be imported at -the top of the test file: - -``from .helpers import validates`` - -Then, your test function should be decorated like this: - -.. code-block:: python - - @validates("R-123456", - "R-123457") # these requirement IDs should come from the VNFRQTS project - def test_my_new_requirement(): - -This decorator is used at the end of the test suite execution to generate a -report that includes the requirements that were violated. If a test is not -decorated it is unclear what the reason for a failure is, and the -implication is that the test is not needed. - -Test Parameters -~~~~~~~~~~~~~~~ - -Each test should be parameterized based on what artifact is being validated. -Available parameters are enumerated in -``/validation-scripts/ice_validator/tests/parameterizers.py``. Below is a description -of the most commonly used: - - - ``heat_template``: parameter is the full path name for a file with the - extenstion ``.yaml`` or ``.yml``, - if the file also has a corresponding file with the same name but - extension ``.env``. - - ``yaml_file``: parameter is the full path name for a file with the - extenstion ``.yaml`` or ``.yml`` - - ``yaml_files``: parameter is a list of all files with the extenstion - ``.yaml`` or ``.yml``. - - ``volume_template``: parameter is the full path name for a file name - that ends with ``_volume`` and the extension ``.yaml`` or ``.yml``. - -There are many others that can also be used, check ``parameterizers.py`` for -the full list. - -The parameter that you decide to use determines how many times a test is -executed, and what data is available to validate. For example, if the -test suite is executed against a directory with 10 ``.yaml`` files, and -a test is using the parameter ``yaml_file``, the test will be executed -once for each file, for a total of 10 executions. If the parameter -``yaml_files`` (note the plural) is used instead, the test will -only execute once. - -Here's an example for how to parameterize a test: - -.. code-block:: python - - @validates("R-123456", - "R-123457") - def test_my_new_requirement(yaml_file): # this test will execute for each .yaml or .yml - -Collecting Failures -~~~~~~~~~~~~~~~~~~~ - -To raise a violation to ``pytest`` to be collected and included on the final -violation report, use the ``assert`` statement. Example: - -.. code-block:: python - - @validates("R-123456", - "R-123457") - def test_my_new_requirement(yaml_file): - my test logic - ... - ... - ... - - assert not failure_condition, error_message - -As one of the VVP priorities is User Comprehension, the ``error_message`` -should be readable and include helpful information for triaging the failure, -such as the ``yaml_file``, the parameter the test was checking, etc... - -If the assert statement fails, the failure is collected by ``pytest``, and the -associated requirements and error_message are included in the final report. - -Optional: Pytest Markers and Validation Categories -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The VVP test suite has the concept of a ``base`` test. These are used as -sanity tests and are executed before the other tests, and if they fail the -test suite execution is halted. If you are writing a ``base`` test, mark your -test like this: - -.. code-block:: python - - import pytest - - @pytest.mark.base # this is the base test marker - @validates("R-123456") - def test_my_new_requirement(): - -The VVP test suite also has the concept of a ``category`` to -define what additional set of optional tests to execute. The way it works -is by using ``categories`` decorator. - -By default, all ``base`` tests and tests with no category are executed. -If you want an additional category to run, pass the command line argument: - -``--category=`` - -This will execute all ``base`` tests, non-marked tests, -and tests marked like the following: - -.. code-block:: python - - import pytest - - @categories("") # substitue with the category name - @validates("R-123456") - def test_my_new_requirement(): - -This should be used sparingly, and in practice consider reviewing a requirement -with the VNF Requirements team before adding a test to a category. - -Self-Test Suite -~~~~~~~~~~~~~~~ - -The VVP test suite includes an extensive self-test suite. This can be -executed by running - -``$ pytest --self-test tests/`` - -This self test suite is used as a check for any new or modified tests. - -If you are adding a new test, a new self-test ``fixture`` **MUST** be created -in the directory ``/validation-scripts/ice_validator/tests/fixtures``. The -directory should be named identical to the new python file (without the ``.py`` -extension), and it should contain 2 subdirectories: ``pass`` and ``fail``. - -These directories should include heat templates that pass and fail the new test. -For Example, if I have created a new test called ``test_my_new_requirement.py`` -, I should create: - -``/validation-scripts/ice_validator/tests/fixtures/test_my_new_requirement/pass`` -``/validation-scripts/ice_validator/tests/fixtures/test_my_new_requirement/pass/pass.yaml`` -``/validation-scripts/ice_validator/tests/fixtures/test_my_new_requirement/fail`` -``/validation-scripts/ice_validator/tests/fixtures/test_my_new_requirement/fail/fail.yaml`` - -When executing the self-test suite, the templates in these folders are -expected to pass and fail, respectively, **ONLY** for the corresponding test. -They don't need to pass the whole test suite. +Template Requirements, visit the `Heat Requirements `__. +Instructions for installing, using, and contributing to the +VVP Validation Scripts project are located `here `__. \ No newline at end of file -- 2.16.6