TESTING.adoc: add a document that describes how ELBE is tested - #486
TESTING.adoc: add a document that describes how ELBE is tested#486kanavin wants to merge 1 commit into
Conversation
Note that I didn't try to make the formatting perfect (or even correct): for a start I wanted to write the content down, so we can get that right first. Signed-off-by: Alexander Kanavin <alex@linutronix.de>
|
Before looking at the contents, I think instead of adoc we should use ReST. |
|
|
||
| To redirect output to somewhere else, use the `--basetemp` option: | ||
|
|
||
| $ pytest --basetemp=$HOME/elbe-tests/ |
There was a problem hiding this comment.
I have also fought with this aspect for some time and it ended up in this patch suggestion:
| not recommended in local development as it will take many hours. It's better to run | ||
| particular relevant slow tests, for example: | ||
|
|
||
| $ pytest -s -k "test_base_extended_build[simple-validation-image.xml]" --runslow --elbe-use-initvm=existing |
There was a problem hiding this comment.
I think this --elbe-use-initvm=existing should be explained a little more. I repeatedly ran against walls with this. When I skipped it, I often had the issue that it collides with the initvm that I have already set up (is it even possible to have more than one ELBE initvm at the same time?). When I used the existing VM, I repeatedly forgot to rebuild the VM properly after doing changes to ELBE that were then not integrated into the VM. There is also the elbe initvm sync command that should help to avoid rebuilding the initvm from scratch, but it failed for strange reasons I did not want to dig deeper into. I ended up spawning fresh VMs with support for nested VMs inside, so I could cleanly rebuild the initvm inside, for every test run to sidestep all these issues, but I think these issues deserve more documentation.
There was a problem hiding this comment.
is it even possible to have more than one ELBE initvm at the same time?
Yes, but you need to fiddle with ports. Theoretically the tests could do this automatically though.
There was a problem hiding this comment.
There is also the elbe initvm sync command that should help to avoid rebuilding the initvm from scratch
This should be mentioned for sure.
There was a problem hiding this comment.
but it failed for strange reasons I did not want to dig deeper into
If this happens again, please let me know.
|
|
||
| On the top level, there are pytest test cases, defined in `elbepack/tests/test_xml.py`. | ||
|
|
||
| They're visible in list of tests provided by `pytest --co`: |
There was a problem hiding this comment.
Let's use the long-form arguments. pytest --collect-only
| @pytest.mark.slow | ||
| @pytest.mark.parametrize('check_build', ('schema', 'cdrom', 'img', 'sdk')) | ||
| def test_simple_build(simple_build, check_build): | ||
| run_elbe_subcommand(['check-build', check_build, simple_build]) |
There was a problem hiding this comment.
Please no copy of source code. If we need to show source code it should be included from the actual file.
| <action><login>root</login></action> | ||
| </check> | ||
| <check-script location="simple-validation-image-test.py" /> | ||
| </check-image-list> |
There was a problem hiding this comment.
We (should) have dedicated docs for elbe check-build, let's link to that instead.
| <Function test_check_updates[simple-validation-image.xml]> | ||
| <Function test_base_extended_build[simple-validation-image.xml]> | ||
|
|
||
| The same set of tests is provided for every `tests\simple-*.xml` image. |
There was a problem hiding this comment.
Forward slash in the file path.
| not recommended in local development as it will take many hours. It's better to run | ||
| particular relevant slow tests, for example: | ||
|
|
||
| $ pytest -s -k "test_base_extended_build[simple-validation-image.xml]" --runslow --elbe-use-initvm=existing |
There was a problem hiding this comment.
We should explain --elbe-use-initvm somewhere.
| not recommended in local development as it will take many hours. It's better to run | ||
| particular relevant slow tests, for example: | ||
|
|
||
| $ pytest -s -k "test_base_extended_build[simple-validation-image.xml]" --runslow --elbe-use-initvm=existing |
There was a problem hiding this comment.
is it even possible to have more than one ELBE initvm at the same time?
Yes, but you need to fiddle with ports. Theoretically the tests could do this automatically though.
| not recommended in local development as it will take many hours. It's better to run | ||
| particular relevant slow tests, for example: | ||
|
|
||
| $ pytest -s -k "test_base_extended_build[simple-validation-image.xml]" --runslow --elbe-use-initvm=existing |
There was a problem hiding this comment.
There is also the elbe initvm sync command that should help to avoid rebuilding the initvm from scratch
This should be mentioned for sure.
| not recommended in local development as it will take many hours. It's better to run | ||
| particular relevant slow tests, for example: | ||
|
|
||
| $ pytest -s -k "test_base_extended_build[simple-validation-image.xml]" --runslow --elbe-use-initvm=existing |
There was a problem hiding this comment.
but it failed for strange reasons I did not want to dig deeper into
If this happens again, please let me know.
3 participants
Note that I didn't try to make the formatting perfect (or even basically correct): for a start I wanted to write the content down, so we can get that right first.
@koalo Please take a look too.