v0.13

A lot of changes since v0.12, which was tagged over a year ago on 9th April 2018.

Abdul Haleem (2):
      start_debugvtty_session: function to create debug vtty session
      ASM: function to enable error injection policy

Abhishek Goel (1):
      OpTestEM : Monitor time as well as usage to test cpuidle driver

Alistair Popple (2):
      OpTestConfiguration.py: Update Skiboot permission check for Mambo
      OpTestConfiguration.py: Update vmlinux permission check for Mambo

Andrew Donnellan (1):
      OpTestSystem: Stop IPL when skiboot abort detected

Aneesh Kumar K.V (2):
      op-test: Move SOL monitor thread to seperate file
      op-teste: Add a separate console log for SOL monitor

Balamuruhan S (5):
      OpTestInstallUtil: introduce API to update kernel cmdline
      OpTestConfiguration: support for kernelcmdgroup to add/remove kernel cmd args
      OpTestKernelArg: test to add/remove kernel commandline and reboot
      op-test: include fedora to be considered for installation
      rhel.ks: add `--ignoremissing` avoid interruption if packages are missing

Brenda Lee (4):
      Initial PCI link problem detection
      PCI link problem detection
      PCI link problem detection
      Initial cpu idle states host enhancement

Deb McLemore (87):
      OpTestSystem: Detect machine state
      OpTestSystem: Connection enhancements
      OpTestSystem: Increase loop_max for SBE Update
      OpTestSystem: Cleanup and consistency for ssh
      OpTestHost: Fix host_check_command arguments
      OpTestHMIHandling: Recover machine from failures
      OpTestSystem: Cleanup obsolete code
      OpTestLogging: Add python logger infrastructure
      OpTestConfiguration: Add AES and Hostlocker Integration
      OpTestIPMI: Fix syntax for module import usage
      Exceptions: Fix format style
      OpTestUtil: Exception improvements
      OpTestConfiguration: FSP IPMITool usage
      DeviceTreeValidation: Use host console for methods
      OpTestConfiguration: Exception handling cleanup
      OpTestSystem: Improve data available on crash
      Variable names for expect pty objects
      OpTestQemu: Fix variable name missed pty
      README: Add python modules
      OpTestMambo: Integrate Mambo
      DeviceTreeValidation: Fix host_get_proc_gen method usage
      DeviceTreeValidation: Diff and compare enhancements
      OPTestHostboot: Add tests to check for platform errors
      DeviceTreeValidation: GoTo proper state for tests
      FWTS: Catch QEMU exception to skip test
      OpTestPNOR: Fix KeyError for flags
      OpTestOpenBMC: Convert to Requests
      IplParams: Skip for QEMU/Mambo
      KernelLog: Filter out urandom_read
      PciSlotLocCodes: Skip for QEMU/Mambo
      README: Add Quick Start section
      AES/Hostlocker: Allow ability to wait for reservations
      InstallHost: Unexpected configuration
      OpTestUtil: Cleanup utility invocations
      OpTestUtil: AES SQL error handling
      PciSlotLocCodes: Modify tests to query the DT
      KernelLog: Add new filter
      OpTestUtil: Dump Firmware Versions Tested
      OpTestUtil: Filter firmware versions message
      PciSlotLocCodes: Skip NPU for slot label checks
      op-test: Add argv to debug logs
      OpTestOpenBMC: Disable one_time boot option
      README: Add python modules for unittest and xml
      op-test: Temporary test removals
      op-test: Fix exit_code
      op-test: only_flash INFO message
      OpTestInbandIPMI: Log failure info to debug logs
      OpTestFlash: Fail test when unable to open tar
      OpTestUtil: Generalize the lsprop filtering
      OpTestSystem: Reduce kill_cord ceiling
      OpTestUtil: Exception handling
      README: Remove old xmlrunner
      op-test: Provide helpful info when suite not found
      BasicIPL: import pexpect for EOF
      OpTestFlash: Fail PNOR/BMC flash when file not there
      OpTestConfiguration: Update AES Server
      README: Add qemu needed packages
      gcov: Skip test if not a gcov build
      OpTestInbandIPMI: Disable dcmi power reading
      op-test: Add test suites for per-commit and pull-request
      OpTestSystem: Recover lost sys_pty
      IplParams: Skip test if non-GA processors
      OpTestFastReboot: Remove devicetree strings from format
      FWTS: Add PATH for RHEL
      OpTestUtil: Add log location at end of run
      OpTestPrdDaemon: Add some recovery logic
      OpTestOCC: Catch exceptions to fail test
      OpTestUtil: Enable for bmcweb
      SbePassThrough: Verify opal-prd running
      OpTestSystem: Aide PTY tracing
      OpTestOpenBMC: Download dumps to logdir
      OpTestPNOR: Add retry
      OpTestInbandIPMI: Pass on SEL no entries
      OpTestHMIHandling: opal-gard interactive support
      IplParams: enable for dd2.3
      OpTestInstall: Network config and timing
      OpTestCronus: Add Cronus support
      OpTestConfiguration: logfile abstraction cleanup
      DeviceTreeValidation: Add lsprop -R and debug info
      OpTestIPMI: Aide PTY tracing
      OpTestPCI: Refactor classes for reuse
      OpTestSystem: Simplify kernel grep
      README: Clarify skiboot cloning steps
      OpTestConstants: Extend HTTP RETRY
      OpTestQemu: Cleanup temp disk file
      OpTestConfiguration: Log git level
      Catch git describe failure

Harish (5):
      Fix OpTestThread: Minor fixes for OpSolMonitorThread3
      RunHostTest: Monitor console on running host commands
      Fix re-connecting console thread
      Fix thread monitoring in RunHostTest
      Support for PowerVM based HMC console

Jeremy Kerr (5):
      README: Add link to documentation
      op-test: Fix indenting on try/except blocks
      op-test: refactor conditional import of xmlrunner
      op-test: Add --failfast option
      qemu: include reason for pexpect spawn failures

Joel Stanley (4):
      Put error messages on one line
      TestConfiguration: Correct flash-skiboot error message
      Add check for the MMU type under petitboot
      Add PetitbootMMU to Skiroot, Mambo and Qemu suites

Mahesh Salgaonkar (1):
      OpTestHMIhandling: Fix the HMI handling for Power8

Oliver O'Halloran (1):
      util: Make hostlocker config parsing more robust

Paidipeddi Pridhiviraj (23):
      Merge pull request #230 from huntbag/master-bh
      Merge pull request #234 from sathnaga/host_test
      Merge pull request #249 from sammj/autoinstall
      Merge pull request #259 from ping678/master
      Merge pull request #260 from vaibhav92/config-read-order
      Update OpTestHost.py
      Merge pull request #280 from lbrenda/master
      Merge pull request #282 from debmc/check_command
      Merge pull request #283 from debmc/hmi
      Merge pull request #284 from debmc/system_cleanup
      Merge pull request #287 from debmc/debs_logging
      Merge pull request #288 from pridhiviraj/newline_fix
      Merge pull request #289 from pridhiviraj/loggers_change
      Merge pull request #291 from ajdlinux/master
      Merge pull request #295 from sathnaga/copy_files_fixup
      Merge pull request #303 from vaibhav92/fix_capi_probe
      Merge pull request #236 from pridhiviraj/em_stress
      Merge pull request #301 from pridhiviraj/configure_ip
      Merge pull request #312 from harish-24/monitor_test
      Merge pull request #285 from debmc/pty
      Merge pull request #314 from debmc/fixpty
      Merge pull request #315 from PavamanSubramaniyam/fw_features_fix
      Merge pull request #313 from balamuruhans/kernel_param

Pavaman Subramaniyam (5):
      Fixed the device tree name from fw-ltpter-serialized to fw-ltptr-serialized
      Enabling EPOW tests for executing on P9 platform
      Fixing the spelling mistake from loge.error to log.error
      Adding power limit support for P9 platforms
      Adding support for Power 9 Boston and Witherspoon CORAL Hardware with DD2.0 and DD2.1 processors

Philippe Bergheaud (9):
      OpTestCAPI.py: stress memcpy afu with many processes
      OpTestHost: discriminate between OpenCAPI and CAPI cards.
      Add OpTestOpenCAPI testcase
      Add opencapi test suite
      host_clone_libocxl: Do not ignore certificate validation
      Fix error reporting of capi/opencapi test cases
      OpTestCAPI: stress memcpy afu to the limit with -p0
      OpTestCAPI: new memcpy test MemCpyAFUReallocTest
      OpTestCAPI: new memcpy test MemCpyAFUIrqTest

Pridhiviraj Paidipeddi (51):
      OpTestSSH/IPMI: Fix printing command two times
      OpTestEM: Add checks for WoF support by looking at DT properties
      OpTestEM: Fix range of frequencies to verify in performance governor.
      SecureBoot: skip tests on systems which don't have support for secureboot
      OpTestEM: Fix tearDown properly
      Add testcase to verify fw-feature's which set during IPL time
      OpTestOCC: Enable for all OpenPower platforms
      OPexpect: Add watchdog hard LOCKUP error messages
      OPexpect: Add HW core checkstop error condition
      OpTestPrdDriver: update IPOLL_MASK_REGISTER_CONTENT for P9
      OpTestHMIHandling: make use of PlatformError exception incase of xstop
      OpTestEM: Remove redundant tear_down function.
      Add test framework flow functional diagram.
      I2C: Fix failure of i2c adapter check in petit kernel
      SecureBoot: Remove the check for secure mode for KT test
      SecureBoot: Remove redundant goto poweroff state
      OpTestSSH: Raise SSH disconnect exception incase of reboot.
      conf: Add UNKNOWN_BAD choice for machine state argument
      conf: Fix failure of list suites option
      OpTestSystem: Improve petitboot_exit_to_shell
      BootTorture: Make boot torture tests more reliable to run
      OpTestSystem: Exit the test when BMC failed to power off the system
      RunHostTest: Fix failure of commands by stripping out the command
      conf: Enable t...

v0.12

Changes since v0.11:

Andrew Jeffery (1):

• OpTestPNOR: Clean up exception handling when flashing PAYLOAD

Deb McLemore (1):

• Enable OpTestSSH for sudo

Mahesh Salgaonkar (2):

• Optimize the common code that gathers chip id and core id.
• hmi: ignore PROC_RECV_ERROR_MASKED hmi test for p9

Paidipeddi Pridhiviraj (2):

• Merge pull request #215 from amboar/payload-exception-handling
• Merge pull request #210 from PraveenPenguin/fix_issue

Praveen K Pandey (2):

• Fixed timeout issue on ubuntu installation
• Fixed issue in installation when user miss host user name

Pridhiviraj Paidipeddi (35):

• OpTestFlash: Ensure flash tests need specific tools.
• This patch is ported from the below PR. Which is the initial work done by Philippe Bergheaud. #94
• OpTestIPMI: import OpTestSystem as set_system_to_UNKNOWN need it.
• Fix run_command API in OpTestSSH
• RestAPI: Modify and enhance rest api test suite
• Fix proc gen issues and color argument failure for dmesg
• Add OPAL sysfs tests.
• OpTestOpenBMC: Add TPM policy setting API.
• OpTestIPMI: Add set/clear TPM Required sensor setting
• OpTestFlash: Add flashing support for other PNOR partitions as well
• OpTestFlash: Support pflash based PNOR update for SMC systems
• OpTestFlash: Force BIOS Golden side sensor to primary side
• Add secureboot IPL and Key Transition tests
• OpTestMtdPnorDriver: Modify new fcp tool related compilation changes
• OPexpect: Fix timeout failures in some cases
• OPexpect: Fix set_system function
• OpTestHost: Fix host_enable_kdump_service with systemctl
• OpTestKernel: Add error conditions for kdump kernel boot
• OpTestOpenBMC: Remove hard coded user/password values for RestAPI login
• OpTestPNOR: Run this test on Host as well.
• SbePassThrough: Fix proc generation issue
• SecureBoot: Add testcase for testing invalid container keys
• Add IPL and OPAL level trustedboot testcases
• SecureBoot: Fix partition list for each proc gen
• SecureBoot: Enhances tests to easy hnadling of key transitions
• OpTestHost: Fix failure of set_system function call back
• OpTestRTCdriver: Remove hwclock_compare function as it no longer supports.
• OpTestFlash: Fix redundant running of sdr list command twice
• SecureBoot: Fix DT properties check for P8 vs P9 In P8 platforms ibm,secureboot-v1 uses hash-algo and in P9 platforms ibm,secureboot-v2 uses hw-key-hash-size. So this patch fixes this accordingly.
• OpTestHMIHandling: ensure disable idle states when cpupower fails to do
• OpTestPNOR: Fix failure of parsing partitions when flags are optional
• OpenBMC: Fix BMC reboot path and login failures.
• TrustedBoot: Fix a broken testsuite failure because of enabled TPM policy
• OpalSysfsTests: Fix failure of set power-cap on SMC systems
• OpTestEM: Add testcase for boost frequencies(WoF)

Samuel Mendoza-Jonas (2):

• OpTestSSH: Make an SSH helper common to all platforms
• OpTestSSH: Allow user to check SSH host keys

Satheesh Rajendran (5):

• Fallback to alternate method to determine ip for boxes with older nc utility
• Add commandline params needed for OS deploy
• Add install library and update Ubuntu install test
• Add support for RHEL OS installation
• Add support for HostOS Installation

Stewart Smith (43):

• DeviceTreeValidation: SSH to host before trying to run commands
• DeviceTreeValidation: fix get_proc_gen for 'POWER9, altivec enabled'
• KernelLog: filter out 'mpt3sas_cm1' not just cm0
• OpTestEM: At end of test, always re-enable CPU frequencies and idle states
• PciSlotLocCodes: allow grep to find 0 lines
• KernelLog: ignore more spurious errors, --color=NEVER EVER EVR
• OpTestFlash/OpenBMC: there can be more than one image
• OpTestIPMI: fix set_system_to_UNKNOWN
• OpTestSSH: fix throwing CommandFailed
• OpTestPNOR: Check if PAYLOAD is marked READONLY
• OpTestFastReboot: reboot twice, fix cpuinfo parsing
• gcov: initial skiboot gcov support
• qemu: don't print repr() each response on console
• OpalSysfsTests: add to default suites
• OpalSysfsTests: Use --color=never for ls
• Add recoverable HMIs to default skiroot test suite
• Merge Pridhiviraj's PNOR flashing and Secure Boot tests
• HMI suite should be run on host, not skiroot
• Merge pull request #211 from maheshsal/mahesh_wip
• Merge remote-tracking branch 'pridhiviraj/force_golden_side'
• OPexpect: find OPAL logging errors
• OpalSysfsTests: stty cols 300 before running test
• OpTestHMIHandling: you can only ping the host once it's booted
• OpTestSystem: Detect IPMI shutdown req to Hostboot
• OpenBMC: try logging/DeleteAll first, before deleting one-by-one by ID
• OpTestSystem: Set state to UNKNOWN on Hostboot IPMI shutdown
• OpTestInbandIPMI: Modify for OpenBMC based on OpenBMC dev feedback
• Merge pull request #214 from pridhiviraj/new
• Merge pull request #218 from maheshsal/mahesh_fixes
• Merge remote-tracking branch 'debmc/sudo'
• InstallUbuntu: fix for qemu use case
• OpTestInbandIPMI/OpenBMC: bootdev diag is unimplemented on OpenBMC
• OpTestEM/cpu_idle_states: use hex mask for taskset, compatible with busybox
• Merge pull request #220 from pridhiviraj/sb_fix
• KernelLog: skip xive interrupt level mismatch
• Merge pull request #219 from pridhiviraj/pflash_parse_fix
• Merge pull request #223 from pridhiviraj/hmi_fix
• We don't have Centaurs on P9
• FWTS: Ignore false failure due to FWTS bug r.e. boost frequencies
• FWTS: skip strange opal-prd.service problems
• Merge pull request #227 from pridhiviraj/tb_fix
• Merge remote-tracking branch 'pridhiviraj/openbmc_fixes'
• KernelLog: ignore systemd-journald file corrupted message

op-test-framework v0.11

Most notable in this release is the start of documentation both for end users and developers of op-test. See it up at https://open-power.github.io/op-test-framework/

Changes since v0.10:

Nagaraju Goruganti (1):

• Changed few operations to use new dbus interface /xyz/openbmc_project/

Pridhiviraj Paidipeddi (31):

• Fix OPexpect with respect to old pexpect versions.
• Fix skipping RestAPI testcase on non OpenBMC based systems.
• Fix failure of getting the core id's for GARDed cores.
• BMC Firmware upgrade tests for both OpenBMC and SMC systems.
• Fix wait for BMC ready state path.
• Change the order of OPAL/kernel error log tests
• EEH Test improvements
• OpenBMC: Add FFDC dump cature related API.
• OPexpect: detect kernel Oops
• OpTestEEH: Add a method to skip known PHB's/PE's from user.
• op-test: Add BMC FW image flash test to default flash test suite.
• OpenBMC: disable field mode before flashing host firmware(OPAL).
• Fix getting list of core ids for each chip.
• op-test: run RestAPI testcase after defualt & IPL suites
• Sbe passthrough testcase.
• Device Tree Validation tests.
• OpTestEM: Test all CPU Pstates instead of just random number.
• SLW: Run test properly on P8 vs P9.
• OpTestEM: Add performance/powersave governor change tests.
• FastReboot: Enable it on Power9.
• BasicIPL: Terminate the console after BMC reboot.
• OpTestFlash: Fix HPM Code update scripts for P8 platforms.
• OpTestSystem: Increase IPL petitboot timeout to 1000 sec's.
• OpTestEEH: Add location code check after a PE EEH error injection
• BootTorture: Add OS soft reboot torture test.
• Add testcase to check for DeviceTree Warnings and errors.
• Test PCI device Slot location codes.
• OpTestConfiguration: Fix SMCIPMI configuration.
• OpTestIPMILockMode: Fix the whitelist of SMC system for IPMI Lock/Unlock feature. Due to the P9 mbox design change and also due to other SMC related changes and optional commands not all AMI supported whitelist command's won't work on SMC systems. This patch makes the whitelist's properly for both AMI and SMC systems accordingly.
• Fix BMC FW image flash test according to OpenBMC Implementation.
• RestAPI: Add OCC active check and power cap related API's.

Stewart Smith (29):

• qemu: boot with a disk attached
• OpTestSystem: Fix PETITBOOT_SHELL to PETITBOOT state transition
• OpenBMC: fix bug with sorting Host from BMC images
• OpenBMC: The BMC may take a moment to make images appear after upload. Wait for it.
• OPexpect: detect kernel hard lockup
• InstallUbuntu: New test to install Ubuntu
• OPexpect: capture reboot due to Platform error
• OPexpect: fix missing comma
• KernelLog: filter out some false positives
• AT24driver: don't use i2cdump, doesn't do the right thing for reading EEPROMs
• KernelLog: '--color=never' to prevent systemd from being 'helpful'
• KernelLog: Ignore harmless kernel 'reserved mem not enough space' warning
• Fix trailing whitespace
• Initial Documentation efforts
• Deal with IPL/boot failures when booting OS
• travis/docs: don't keep history of gh-pages
• testcases/KernelLog: Add more false-positives to exclude from check
• OpTestEM: Add test specifically entering/exiting each idle state
• OpTestOpenBMC: use new method of deleting software images
• doc: Add firmware flashing docs
• qemu: further work towards full testing in Qemu
• doc: Add more (brief) documentation on some Exceptions
• doc: Better document OPexpect
• doc: Document OpTestError as deprecated
• doc: make docs pretty, link to GitHub
• doc: futher document how Qemu support works
• doc: use ini file syntax highlighting where needed
• doc: highlight command as bash
• doc: document command line options

v0.10

This release brings a lot of improvements. There is better support for OpenBMC, FSP and qemu platforms for flashing firmware and detecting errors earlier.

We also add the ability to store op-test parameters in a configuration file, so you can have a config file globally or per machine.

Jason Albert (2):

• Support for setting output location
• Fix for ssl unverif context not supported in python < 2.7.9

Pridhiviraj Paidipeddi (16):

• Add new code update method for OpenBMC systems.
• At24driver: check repeated reads of EEPROMs
• BMC Reset Torture Vs Host IPL Torture tests.
• Improvements for getting core ids.
• Add PNOR and OPAL Lids Flash support for Supermicro systems.
• Add new helpers for proc_gen, core_count and smt.
• Add CPU Hotplug testcase.
• Fix surv message check after fsp rr.
• Fix LID's flashing test on FSP systems.
• OpTestIPMILockMode: Enable the test on boston platform.
• OpTestOOBIPMI: Fixes the testcase to pass on SMC systems.
• OpalUtils: Fix the xscom write to work on P9
• OpTestEM: Log and assert the cpuidle driver disable failure message.
• Enable HMI tests on P9.
• Fix output dir for gathering opal and kernel logs.
• Testcase for OpenPower Gard Interfaces.

Russell Currey (1):

• Explicitly use python2 instead of /usr/bin/python

Stewart Smith (47):

• OpTestFlash: more reliably detect old/new OpenBMC PNOR update mechanism
• OpTestBMC: run rsync with -P to show progress
• OpTestOpenBMC: Parse json rather than use regex
• OpenBMC flashing fixes
• OpenBMC: new method of telling Chassis power status
• OpenBMC: Fix skiboot/skiroot flashing
• OpTestPrdDriver: we don't need git,gcc on host
• op-test: use faulthandler to enable SIGUSR1 to force backtrace
• OpenBMC: fixes for OpenBMC API ambiguity and bugs
• OpenBMC flashing: Hardcode BOOTKERNEL padding
• OpenBMC: Correctly list software on BMC
• OpenBMC: software IDs are lies
• OpTestOpenBMC: only monitor Host status, not chassis
• OpTestIPMI: SKIP rather than Fail on no USB IPMI
• OpTestInbandIPMI: fix error message for IPMI boot dev test
• OpTestFlash: OpalLidsFLASH: transfer pflash for flashing individual partitions
• Add SMC BMC type
• OpTestInbandIPMI: Only fail sensor tests if sensor is meant to be present
• OpTestInbandIPMI: be more flexible with dcmi results
• Merge pull request #173 from ruscur/python2
• Merge pull request #163 from hegdevasant/prd-daemon-fix
• OpTestIPIM: Improve error reporting of errors during IPL
• OpenBMC: Also remove GUARD records
• BootTorture: head/tail opal log, only check PCI if --lspci specified
• op-test: support --no-flash and --noflash
• testcases/OpalMsglog: enable the possibility of success
• Merge pull request #176 from hegdevasant/bugfix
• Console.ControlC: Wait 2 seconds rather than 0.2
• IPMI: skip rather than fail on absent USB interface
• OpenBMC: fix code update for > v1.99.10-0-r13 due to introduciton of 'functional' entry in software/
• OpenBMC: fail quick on image Activation failure
• OpenBMC: restart mboxd before applying PNOR overrides
• Filter out control characters when logging to console
• Add --config-file support
• fixup for no config file
• Don't echo 'hello World' to console on start
• Qemu: console filtering/logging for qemu
• testcases/DPO: fail fast if ipmi device doesn't exist
• testcases/Console: aid debugging ability, print expected line number count
• qemu: s/\r\r\n/\n/ on qemu console
• Fix --help to print full help
• op-test: Add --config-file to --help
• whitespace fix
• OpTestSystem: count down waiting for petitboot...
• OPexpect: a pexpect wrapper that catches common OpenPOWER failures
• OpenBMC: Try hard power off
• OpenBMC: Properly handle timed-out commands running on host

Vasant Hegde (5):

• OpenBMC: Fix pflash command detection logic
• testcase/OpTestPrdDaemon: Enable on P9 FSP system
• testcase/OpTestPrdDaemon: Improve testcase
• OpTestFlash: Use 'cp -f' to copy LID to FSP
• testcases/Console: Increase timeout value

v0.9

Jason Albert (2):

• Remove duplicate cmdline args print in addons conditional
• Rename addon AddBMCType function to addBMCType

Pridhiviraj Paidipeddi (10):

• FW Flash Testcase for FSP systems.
• HMIHandling: simplify IPL monitoring path.
• OpTestPrdDaemon: Fix the command order.
• I2C: Fix verification of list of chips.
• Enhance and fix rest API for OOB Management.
• BootProgress: retrieve the boot progress and print.
• Sel: enable sel clear/list during boot.
• testRestAPI: Log Rest command output only in full testcase.
• RestAPI: Change /org/openbmc/ to /xyz/openbmc_project/ interfaces
• OPAL Lids flashing support in FSP/AMI/OpenBMC systems.

Stewart Smith (19):

• SSHConnection: add run_command_ignore_fail
• I2C: Fail gracefully if no EEPROMs found
• InbandUsbInterface: Skip on OpenBMC
• qemu: fix build failure
• op-test: fix exit code on test success/failure
• op-test: add --noflash and --only-flash options
• OpTestFlash: don't boot to Petitboot as part of flashing
• Update README for flashing
• op-test: Flash whole PNOR before individual partitions
• OpTestIPMI: if SOL connection isn't alive, still do state machine ops
• SEL: only write SELs to ffdc dir if dir is specified
• Don't require flashing whole PNOR
• op-test: increase verbosity with XMLTestRunner
• Improve wait_for_petitboot logic
• IPMI: better retry on 'SOL Disconnected on BMC', up to 'mc reset cold'
• OpTestFlash: don't fail on not flashing FSP
• OpTestSystem: don't cancel autoboot on petitboot screen
• OpTestIPMI: Use 'chassis power status' as fallback to Host status sensor
• OpTestFlash: copy pflash to BMC (if needed)

Suraj Jitindar Singh (1):

• test: Add OpTestPNOR to test host pnor access

v0.8

Jason Albert (2):

• New support for addons, allowing for dynamic loading of bmc types
• Support for addons to define additional test suites

Pridhiviraj Paidipeddi (13):

• Testcase for FSP Console Issue(Bug 150765).
• Testcase for pcie link down/timeout errors.
• PCI Test enhancements.
• REST API enablement for OpenBMC systems.
• Enable PNOR & Skiboot lid flash tests on OpenBMC systems.
• BootTorture: Gather kernel/opal errors from logs.
• PCI Driver bind/unbind test.
• PCI Hotplug tests
• testcases/OpTestEEH: Fix some grep & lspci failures.
• Fix system standby state issue in state machine.
• OpTestIPMI: fix close console path.
• OpTestKernel: Test kernel crash in skiroot.
• OpTestInbandUsbInterface: Skip the tests in FSP systems

Stewart Smith (15):

• README: add opal-utils
• testcases/DPO: support FSP system behaviour
• IPMIConsole: improve timeout/unexpected close recovery
• Merge branch 'pci_enhancements' of https://github.com/pridhiviraj/op-test-framework
• KernelLog: allow grepping for errors to fail, 0 errors=success!
• Merge pull request #142 from pridhiviraj/console_bug_150765
• OpTestOpenBMC: Add run_command_ignore_fail
• OpTestOpenBMC: not all OpenBMCs have SEL clear
• testcases: Add Console test
• OpTestOpenBMC: Fix CURL failure error message
• Add known-bugs test suite, with ctrl-c console test
• testcases/Console: add ctrl-z test
• add Console32k test to known-bugs
• Merge branch 'fixes' of https://github.com/pridhiviraj/op-test-framework
• Merge pull request #149 from thejta/addon-suite

v0.7

Pridhiviraj Paidipeddi (10):

• Enhance I2C and AT24 driver tests to run in skiroot.
• Fix run_command API
• Fix some command executions in OpTestPrdDaemon test.
• Add some debug o/p for opal-prd activation.
• Try HPM Update one more time if it fails for the first time.
• IPMI and Console Torture tests.
• Torture tests for opal elog.
• Fix system dump test case by enabling the dump policy.
• Add HIR Torture test for SURV Ack timeout.
• Testcase for getting Kernel Log errors/warnings.

Stewart Smith (10):

• Use sudo to get host root shell (if needed)
• BootTorture: boot 1024 times to petitboot
• BootTorture: verify PCI device detection
• PetitbootDropbearServer: Add missing host_console_unique_prompt()
• Qemu: Preliminary support for qemu simulator
• OpTestPCI: test reading config space
• OpTestSystem: expect the same way as everywhere else.
• OpTestIPMI: Somewhat improve things when 'SOL Disconnected by BMC' occurs
• Merge pull request #139 from pridhiviraj/torture_tests
• Merge pull request #140 from pridhiviraj/sysdump_fixes

v0.6

Improvements to test runner:

• Save the test execution logs in test-reports directory

Changes to machine state machine:

• Update state machine for ssh connection when system goes to off.
• Fix broken state machine for PETITBOOT_SHELL

New/improved tests:

• TOD Corruption tests.
• Enhancements for frozen PE tests.
• Enhance EM tests to run in skiroot.
• OPAL Utils testcase.
• Enhance RTC driver test to work in skiroot.
• Enhance inband ipmi to work in skiroot.
• Add surv ack timeout HIR.
• Enhance system dump testcase.
• test if the opal-prd daemon is always running
• PRD: Abstract SCOMS for P9
• FWTS: fail properly when FWTS not installed

op-test-framework v0.5

New tests:

• FSP Reset Reload Testcases
• OCC reset test for FSP platforms.
• EPOW3 Low and Random Testcases.
• DPO Testcase.

Modified tests:

• OpTestEM: Measure CPU frequency with ppc64_cpu --frequency to verify setting
• Fix BasicIPL tests for FSP
• OpTestFastReboot: Add some torture tests, useful in development

Other changes:

• run_command for SSH and IPMI now has a consistent API
• we now raise exceptions on command failure and other errors
• OpenBMC: Add IPMI to OpenBMC as OpenBMC now supports out of band IPMI

v0.4

OpenPower Test Framework

This is the first tagged release of op-test-framework. We are in a march towards releasing v1.0 near the end of 2017, and will be making regularly tagged releases based on our progress.

As this is the first release, we're starting with a pretty clean slate (although a lot of changes has occurred during development to get to this point).

Current status / Known Issues

As of today, the default test suite should mostly pass on current POWER8 OpenPOWER systems running current op-build. Expect a few failures from current FWTS (which should be addressed in upcoming FWTS releases). We also have a funny thing going on around CAPI cards, where the PCI device presence test may fail due to the list of PCI devices in petitboot and a booted OS with CAPI support being different.

There is preliminary support for OpenBMC based systems (just access to the console, no support for boot/reboot).

There is quite decent support for IBM FSP based systems.

There is also basic support for POWER9 systems, and this op-test-framework release has been run on them. There's a few rough patches still though, and we're not complete enough there yet.

By default, we run the "basic" suite of tests. You can also run the "full" suite, which will take longer and be a bit more complete. Currently, the 'basic' suite is being used to gate any changes to the skiboot component of firmware.

Feedback is very much appreciated, either on the openpower-firmware@lists.ozlabs.org mailing list, or via github issues: https://github.com/open-power/op-test-framework/issues

v0.4 README

Below, is what is currently in README.md, and is a good place to get started.

This repository provides a collection of tools that enable automated testing of
OpenPower systems. The op-test-framework suite is designed to test a machine
largely out of band - that is, it is designed for tests that do things like
power cycle the machine, test booting different configurations. As part of
the op-test-framework, we may run tests on the host itself (such as fwts
and HTX)

The end goal is to have a collection of tests that can be run against any
OpenPower system to validate it's function. The tests are automation/jenkins
ready.

Requirements

This framework runs on most Linux based systems.

You need python 2.7 or greater.

You will also need (recent) ipmiutil - 1.8.15 or above should be adequate.

You will need to run the test suite on a machine that has access to both
the BMC and the host of the machine(s) you're testing.

Preparation

The target system will need to have an OS that can boot. That OS will
need to have several things installed on it.

Target System Requirements

A basic Linux install is assumed.

You MUST have fwts installed. To do this:

sudo apt-get install software-properties-common
sudo add-apt-repository ppa:firmware-testing-team/ppa-fwts-stable
sudo apt-get update
sudo apt-get install fwts

It must also have (package names for Debian/Ubuntu systems):

linux-tools-common linux-tools-generic lm-sensors ipmitool i2c-tools pciutils opal-prd

On RHEL-like systems, package names are:

lm_sensors ipmitool i2c-tools pciutils kernel-tools

From skiboot, you will need the xscom-utils and gard installed:

git clone https://github.com/open-power/skiboot
cd external/xscom-utils
make
sudo make install
cd ../gard
make
sudo make install

Running the tests

./op-test -h

Gets you help on what you can run. You will need to (at a minimum) provide
BMC and host login information. For example, to run the default test suite:

./op-test --bmc-ip bmc.example.com   \
	      --bmc-username sysadmin    \
      --bmc-password superuser   \
      --bmc-usernameipmi ADMIN   \
      --bmc-passwordipmi admin   \
      --host-ip host.example.com \
      --host-user root		 \
      --host-password 1234	 \
      --host-lspci host.example.com-lspci.txt

The default test suite will then run.

To get a list of test suites:

./op-test --list-suites

You cun run one or more suites by using the --run-suite command line option.
For example, you can choose to run tests that are only at the petitboot
command line. By default, the test runner doesn't know what state the machine
is in, so will attempt to turn everything off to get it into a known state.
You can override this initial state with the --machine-state parameter.
You can also run individual tests by using the --run option.

For example:

  ./op-test --bmc-ip bmc.example.com \
  		--bmc-username sysadmin  \
	--bmc-password superuser \
	--bmc-usernameipmi ADMIN \
	--bmc-passwordipmi admin \
	--host-ip host.example.com \
	--host-user root 	   \
	--host-password 1234	   \
	--host-lspci host.example.com-lspci.txt \
	--machine-state PETITBOOT_SHELL \
	--run testcases.OpTestPCI.OpTestPCISkiroot

The above will assume the machine is sitting at the petitboot prompt
and will run the OpTestPCISkiroot test.