1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
2 .. http://creativecommons.org/licenses/by/4.0
3 .. Copyright 2019 AT&T Intellectual Property. All rights reserved.
5 Manual Heat Template Validation
6 ===============================
13 This project contains validation scripts to test
14 that a set of Heat Templates adheres to
15 the ONAP VNF Heat Orchestration Template guidelines.
17 For more information on the ONAP Heat Orchestration
18 Template Guidelines, vist the `Heat Guidelines <https://onap.readthedocs.io/en/latest/submodules/vnfrqts/requirements.git/docs/Chapter5/Heat/index.html>`__
23 The validation scripts project allows performing heat template
24 validation without installing the full VVP platform. The following
25 instructions apply to running these validation scripts in that manner.
31 This software is not platform dependent and can be run in a Windows, Unix or
38 These can be installed using pip (assuming pip is installed) with the command:
40 ``$ pip install -r requirements.txt``
47 To validate Heat templates just run this the command under the folder ``ice_validator``:
49 ``</path/to/validation-scripts/ice_validator>$ pytest --tap-stream --template-directory=<Directory>``
51 where ``<Directory>`` is the full path to a folder containing heat templates.
56 After performing a validation, an output folder will be created.
58 ``/path/to/validation-scripts/ice_validator/output/``
60 This folder will contain a file ``report.html`` which contains a list of all
61 of the ONAP VNF Heat Template Guideline violations. If there are no violations,
62 the report will say ``No validation errors found.``
64 Interpreting the Output
65 #######################
67 The report file will have 4 columns for details about a violation, and one
68 row for each violation. Below contains details about each column.
73 This is the file(s) that contained the violation
78 This shows the test and brief error message from the validation script that
79 contained the violation. There is a ``Full Details`` button to show the
80 complete raw test output. The error message will also contain details
81 about what element is involved with the violation (such as the parameter
82 name, resource id, etc...).
87 This column contains the requirement(s) that each test/violation is
88 mapped to. These requirements are taken directly from the VNF Requirements
89 project Heat Orchestration Template Guidelines section.
95 For some violations, there are pre-defined resolution steps that
96 indicate what action the user should take to resolve the violation.
98 **Note**: Not all violations will have resolution steps. Most violations
99 can be resolved simply by reviewing the requirements that have been violated
100 in the previous column.
105 Before getting started
106 ######################
111 **The objective for the VVP test suite is for each
112 test to directly correlate with at least one requirement in the**
113 `VNF Requirements <https://onap.readthedocs.io/en/latest/submodules/vnfrqts/requirements.git/docs/index.html>`__
114 **project in ONAP. If the test you intend to write doesn't
115 have a corresponding requirement in the VNF Requirements project, consider
116 making a contribution to that project first.**
118 Convenience vs Convention
119 ~~~~~~~~~~~~~~~~~~~~~~~~~
121 There are a lot of ways to write tests. Priorities for the VVP test suite are
126 The test suite is often used by people who don't write code, or people
127 who aren't devoted to writing python validation tests.
129 The output of failed validation tests can be difficult to read, so
130 keep that in mind when you are deciding whether to create another
131 level of abstraction vs having some code duplication or verbose tests.
139 Test files are written in python, and should go into the
140 ``/validation-scripts/ice_validator/tests/`` directory. They should be prefixed
141 with ``test_``. If not, ``pytest`` will not discover your
142 test if you don't follow this convention.
147 Tests are functions defined in the test file, and also must be prefixed with
148 ``test_``. If not, ``pytest`` will not collect them during execution.
151 **test_my_new_requirement_file.py**
153 .. code-block:: python
155 def test_my_new_requirement():
157 Requirement Decorator
158 ~~~~~~~~~~~~~~~~~~~~~
160 Each test function should be decorated with a requirement ID from the
161 VNF Requirements project. The following is required to be imported at
162 the top of the test file:
164 ``from .helpers import validates``
166 Then, your test function should be decorated like this:
168 .. code-block:: python
170 @validates("R-123456",
171 "R-123457") # these requirement IDs should come from the VNFRQTS project
172 def test_my_new_requirement():
174 This decorator is used at the end of the test suite execution to generate a
175 report that includes the requirements that were violated. If a test is not
176 decorated it is unclear what the reason for a failure is, and the
177 implication is that the test is not needed.
182 Each test should be parameterized based on what artifact is being validated.
183 Available parameters are enumerated in
184 ``/validation-scripts/ice_validator/tests/parameterizers.py``. Below is a description
185 of the most commonly used:
187 - ``heat_template``: parameter is the full path name for a file with the
188 extenstion ``.yaml`` or ``.yml``,
189 if the file also has a corresponding file with the same name but
191 - ``yaml_file``: parameter is the full path name for a file with the
192 extenstion ``.yaml`` or ``.yml``
193 - ``yaml_files``: parameter is a list of all files with the extenstion
194 ``.yaml`` or ``.yml``.
195 - ``volume_template``: parameter is the full path name for a file name
196 that ends with ``_volume`` and the extension ``.yaml`` or ``.yml``.
198 There are many others that can also be used, check ``parameterizers.py`` for
201 The parameter that you decide to use determines how many times a test is
202 executed, and what data is available to validate. For example, if the
203 test suite is executed against a directory with 10 ``.yaml`` files, and
204 a test is using the parameter ``yaml_file``, the test will be executed
205 once for each file, for a total of 10 executions. If the parameter
206 ``yaml_files`` (note the plural) is used instead, the test will
209 Here's an example for how to parameterize a test:
211 .. code-block:: python
213 @validates("R-123456",
215 def test_my_new_requirement(yaml_file): # this test will execute for each .yaml or .yml
220 To raise a violation to ``pytest`` to be collected and included on the final
221 violation report, use the ``assert`` statement. Example:
223 .. code-block:: python
225 @validates("R-123456",
227 def test_my_new_requirement(yaml_file):
233 assert not failure_condition, error_message
235 As one of the VVP priorities is User Comprehension, the ``error_message``
236 should be readable and include helpful information for triaging the failure,
237 such as the ``yaml_file``, the parameter the test was checking, etc...
239 If the assert statement fails, the failure is collected by ``pytest``, and the
240 associated requirements and error_message are included in the final report.
242 Optional: Pytest Markers and Validation Categories
243 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
245 The VVP test suite has the concept of a ``base`` test. These are used as
246 sanity tests and are executed before the other tests, and if they fail the
247 test suite execution is halted. If you are writing a ``base`` test, mark your
250 .. code-block:: python
254 @pytest.mark.base # this is the base test marker
255 @validates("R-123456")
256 def test_my_new_requirement():
258 The VVP test suite also has the concept of a ``category`` to
259 define what additional set of optional tests to execute. The way it works
260 is by using ``categories`` decorator.
262 By default, all ``base`` tests and tests with no category are executed.
263 If you want an additional category to run, pass the command line argument:
265 ``--category=<category>``
267 This will execute all ``base`` tests, non-marked tests,
268 and tests marked like the following:
270 .. code-block:: python
274 @categories("<category>") # substitue <category> with the category name
275 @validates("R-123456")
276 def test_my_new_requirement():
278 This should be used sparingly, and in practice consider reviewing a requirement
279 with the VNF Requirements team before adding a test to a category.
284 The VVP test suite includes an extensive self-test suite. This can be
287 ``</path/to/validation-scripts/ice_validator>$ pytest --self-test tests/``
289 This self test suite is used as a check for any new or modified tests.
291 If you are adding a new test, a new self-test ``fixture`` **MUST** be created
292 in the directory ``/validation-scripts/ice_validator/tests/fixtures``. The
293 directory should be named identical to the new python file (without the ``.py``
294 extension), and it should contain 2 subdirectories: ``pass`` and ``fail``.
296 These directories should include heat templates that pass and fail the new test.
297 For Example, if I have created a new test called ``test_my_new_requirement.py``
300 ``/validation-scripts/ice_validator/tests/fixtures/test_my_new_requirement/pass``
301 ``/validation-scripts/ice_validator/tests/fixtures/test_my_new_requirement/pass/pass.yaml``
302 ``/validation-scripts/ice_validator/tests/fixtures/test_my_new_requirement/fail``
303 ``/validation-scripts/ice_validator/tests/fixtures/test_my_new_requirement/fail/fail.yaml``
305 When executing the self-test suite, the templates in these folders are
306 expected to pass and fail, respectively, **ONLY** for the corresponding test.
307 They don't need to pass the whole test suite.