Half Automate Generate Test Script Model

 Half Automate Generate Test Script Model

What I think is that we can write a tool to convert the test script according to the description above testrail. using python tools automatic write python test cases

The benefits are as follows:

1: You do not need to manually create two files c0000_uefi_test.xml and c0000_uefi_test.py

2: Automatically put the platform information in testrail directly in case.xml, and then set some other parameters by default.

3: Automatically decompose the test steps in testrail directly into the case.py file. The steps are clear and clear. Leave two lines for each step and one line for one line. These codes are generated by the tool.

4: If the amount of the case is large, the development time will be saved.

5: The more perfect testrail is, the higher the degree of automation ,If the testrail is perfect enough, it can be written manually without any need.

Implementation

for example case C167

The following data is what I got from the testrail

{u’type_id’: 6, u’refs’: None, u’priority_id’: 4, u’custom_ffvpqaid’: u’MS-2351’, u’created_on’: 1432048007, u’custom_ffvplatform’: [1, 2, 3, 4, 5, 10, 12, 13, 14, 15, 16, 17, 18], u’milestone_id’: 22, u’title’: u’Firmware downgrade to last released firmware set’, u’custom_ffv_automatable’: None, u’created_by’: 36, u’id’: 167, u’custom_fw_srs’: None, u’custom_ffv_need_physical_access’: 2, u’custom_ffvsystype’: [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12], u’custom_ffv_cpu_specific’: [2], u’updated_by’: 388, u’custom_steps_separated’: [{u’content’: u’Enter POST diagnostic menu.\n 1.Press Crtl+C when booting POST to enter POST Login interface.\n 2. Press GO_sox to login.’, u’expected’: u’System should enter POST root menu without any Warning message and Error message.’}, {u’content’: u’Active POST TFTP sevice\n 1. Select “Image Sub-Menu” --> “TFTP Download”’, u’expected’: u’Tip message "TFTP ready to receive image " apeared.’}, {u’content’: u’Upload bin file by POST TFTP Function.\n 1. Run “CMD> TFTP -i IPaddress put bin file” in Dos.’, u’expected’: u’BIOS, POST, BMC and all devices Firmwares be uploaded without any unexpected problem.\n 1. Bin files should be uploaded without any unexpected problem.\n 2.The upload bin files should be older than all system original devices Firmwares.’}, {u’content’: u’Downgrade all firmwares by POST\n 1). Select “Image Sub-Menu” -->“Set Firmware Match Rev.” to force system will update to target version.\n 2). Reset SP and boot to POST.\n 3). Repeat step 2-3.\n 4). Select “System Test Sub-Menu” --> “Autoflash” to start updating firmware. Input “y” when original firmware version is same as the upload firmware version.’, u’expected’: u’Downgrade process should complete without any problem.’}, {u’content’: u’Reboot system and guarantee all updated firmware take effect.\n 1.Select “Reset Controller” to reboot system.\n 2. Press Crtl+C when booting POST to enter POST Login interface.\n 2. Press GO_sox to login.’, u’expected’: u’need to check if two reset happened during force flash FW:\nthe first time is the warm reset, after that it will start flashing BIOS…\nonce done the SP will reset again.’}, {u’content’: u’Check all updated firmware be updated successful.\n 1. Select “Information display” --> “Firmware Inventory” to check the Firmware version\n 2. Select “Resume” --> “Display Resume” to check the Firmware version\n 3. Select “System Test Sub-Menu” --> “Firmware Revision Validate” to check the firmware version.’, u’expected’: u’Verify all firmware elements(BIOS, POST, BMC, SSP, CMD, PLX, BCM, etc) are upgraded to latest revision successfully. Any workaround or warning/fault must be noted.’}, {u’content’: u’Flash all firmware to original version to restore.’, u’expected’: u’’}, {u’content’: u’Repeat step 1-7 to ensure the firmware downgrade under all POR system type can work as expected.’, u’expected’: u’’}], u’custom_ffv_overview’: None, u’section_id’: 3422, u’custom_preconds’: u’Should be able to downgrade firmware set to last released firmware set.\r\nBefore downgrading, make sure there is no known blocking issue on the previous released firmware set to avoid bricking hardware.\r\n\r\n\r\nTest Methodology:\r\n Downgrade to last released firmware set using “Firmware match Rev” and “Autoflash”;\r\n Autoflash from last released firmware set to target firmware set.\r\n Make sure all flashing are done without any error/warning/rev mismatch.’, u’suite_id’: 9, u’estimate_forecast’: u’2h 18m 42s’, u’estimate’: None, u’custom_ffvorigin’: u’TU-8’, u’custom_ffv_target_test_plan’: [3, 4, 7, 8, 9, 10, 14, 15, 16, 17, 18, 19, 21, 22, 25, 27, 32], u’updated_on’: 1558689347, u’template_id’: 1}

we can see it’s a dictionary, so work is easy now.

step1: get data from testrail

step2: check it is uefi or bmc , and get case title , creat file C167_uefi_Firmwaredowngradetolastreleasedfirmwareset.xml C167_uefi_Firmwaredowngradetolastreleasedfirmwareset.py

step3: put default message in C167_uefi_Firmwaredowngradetolastreleasedfirmwareset.xml, get platform message and replace platform in C167_uefi_Firmwaredowngradetolastreleasedfirmwareset.xml

step4: Define the following in the test script:

1: Required import such as:

from case.CBaseCase import CBaseCase

from pub.Const import FAIL, PASS, INFO, BLOCK, DONE, ERROR

2: common interface

class C167_uefi_DowngradeFirmwarelastRelease(CBaseCase):

def __init__(self):
    super(C167_uefi_DowngradeFirmwarelastRelease, self).__init__(self.__class__.__name__)
 
def config(self)
    PASS
 
def test(self)
    pass
 
def deconfig(self)
    pass

3: step messages

we can also get step messages from testrail, Disassemble each step, then leave three lines in the middle of each step, one line is executed as code, one line is checked, one line is blank.

in test function , we will collect some common functions or Modular codes in a library , like this “”“enter post “”” go_to_post() , fuzzy matching Highest score wins, if some steps similar with common function’s description, call it under steps, if can not find , do not call , and check next step , every step there will be a log trackback. the log is useful for automate analysis

def test(self):

# step1: Enter POST diagnostic menu
self.log(INFO, 'Enter POST diagnostic menu')
ret = sp.go_to_post()
    if ret:
        self.result(FAIL, step 1 messages)
 
# step2 :xxxxxxxxxxxxxx
self.log(INFO, 'xxxxxxxxxxxxxx')
ret = xxxxxxxxx
    if ret:
        xxxxxxxxxx

4: autopep8 formats Python code

5: step 1-4 above all automate tools do what he can , next people join it , and do some check and modify and tests

Supplementary explanation:

future maybe could done , it from step 4 , automate run this case if which step is failed, rewrite it from step by select second scores, record failed and pass history for next case’s development. search similar cases (from title and steps), config deconfig do the same

If there is such coding logic in the future machine learning, I am afraid that the last step does not require human beings. It is a bit scary to think about it, but it will come.

Expand thinking

I think this model is useful for every test teams. we can provide a public solution for test teams. Maybe we can still write a patent. if step 3, we add machine learning to do that , it will be better