Linaro runs a battery of tests on the Open Embedded and Android operating systems using a variety of hardware and kernel versions in order to detect kernel regressions. These regressions are reported to member companies and the various upstream communities like linux-stable.

This report is a summary of our activity this week.

Due to some breakage that was witnessed, there was a desire this month to add KVM into the kernel testing mix. KV-165 Investigate adding KVM testing to LKFT

Testing on Open Embedded

  • KV-126 Final testing for upgrade to sumo happening in staging. Looking to upgrade this week.
  • Db410c boot issue showed up, investigating.
  • Bug Status – 62 open bugs
    • 2019-03-06: bpf_test_tcpbpf_user confirmed fixed on mainline thanks to Anders’ patch re: bug 3938
    • 2019-03-05: Bug 4305 – LTP: Juno: 64K page: causing Unable to handle kernel NULL pointer dereference reported to ARM and LDCG
    • 2019-03-04: Anders Roxell reported a use after free with KASAN in next
    • 2019-03-01: Naresh Kamboju reported a kernel warning triggered by bpf test_sock
  • RC Log
    • 2019-03-04
      • 4.9.162, 4.14.105, 4.19.27, 4.20.14
        • Reported no regressions in <24h

Testing on Android

    • Discussion
      • Pixel 3 is starting to boot : https://pastebin.linaro.org/view/bbdf82eb
      • DB845 is following along as well
      • Working with John/YongQin and hikey-linaro kernel branches that then get used by LKFT. Part of what we’ve been bit by in failures is due to changes in Android Common due to post P.
    • Android 9 / P LTS-premerge – 4.4, 4.9, 4.14, 4.19
      • 4.19.26 / HiKey – no regressions
      • 4.14.104 / HiKey –
        • cts-lkft/arm64-v8a.CtsOsTestCases/android.os.cts.StrictModeTest.testNetwork
        • cts-lkft/arm64-v8a.CtsOsTestCases/android.os.cts.StrictModeTest.testUntaggedSocketsHttp
        • cts-lkft/arm64-v8a.CtsOsTestCases/android.os.cts.StrictModeTest.testUntaggedSocketsRaw
        • cts-lkft/armeabi-v7a.CtsOsTestCases/android.os.cts.StrictModeTest.testNetwork
        • cts-lkft/armeabi-v7a.CtsOsTestCases/android.os.cts.StrictModeTest.testUntaggedSocketsHttp
        • cts-lkft/armeabi-v7a.CtsOsTestCases/android.os.cts.StrictModeTest.testUntaggedSocketsRaw
        • cts-lkft/armeabi-v7a.CtsWebkitTestCases/android.webkit.cts.WebViewSslTest.testProceedClientCertRequestKeyWithAndroidKeystoreKey
      • 4.9.161 / HiKey
      • 4.4.176 / HiKey
        • No regressions
        • cts-lkft/arm64-v8a.CtsUsbTests /com.android.cts.usb.TestUsbTest.testUsbSerialReadOnDeviceMatches
          • Seen before but I believe this test is interesting in the context of adb disconnect issues
    • Android 9 / P –  4.4, 4.9, 4.14, 4.19 + HiKey
      • 4.19.23 / HiKey – no new data
      • 4.14.101 / HiKey – no new data
      • 4.9.158 / HiKey – no new data
      • 4.4.174 / HiKey – no new data
    • AOSP-master-tracking –  4.9, 4.14 4.19 / HiKey & 4.14 / X15
      • hi6220-hikey_4.19.23:
      •  cts-lkft-arm64-v8a/arm64-v8a.CtsBluetoothTestCases:
        • android.bluetooth.cts.HearingAidProfileTest.test_getConnectedDevices
        • android.bluetooth.cts.HearingAidProfileTest.test_getDevicesMatchingConnectionStates
      • hi6220-hikey_4.14.101:
      •  cts-lkft-arm64-v8a/arm64-v8a.CtsBluetoothTestCases:
        • android.bluetooth.cts.HearingAidProfileTest.test_getDevicesMatchingConnectionStates
      •  cts-lkft-armeabi-v7a/armeabi-v7a.CtsBluetoothTestCases:
        • android.bluetooth.cts.HearingAidProfileTest.test_getConnectedDevices
        • android.bluetooth.cts.HearingAidProfileTest.test_getConnectionStateChangedIntent
      •  cts-lkft-arm64-v8a/arm64-v8a.CtsLibcoreTestCases:
      •   org.apache.harmony.tests.java.util.regex.MatcherTest.testAllCodePoints_p
      •  cts-lkft-armeabi-v7a/armeabi-v7a.CtsBluetoothTestCases:
        • android.bluetooth.cts.HearingAidProfileTest.test_getConnectionStateChangedIntent
      • hi6220-hikey_4.9.158:
      •  cts-lkft-armeabi-v7a/armeabi-v7a.CtsBluetoothTestCases:
        • android.bluetooth.cts.HearingAidProfileTest.test_getDevicesMatchingConnectionStates
      • x15_4.14.101:
      •  cts-lkft-armeabi-v7a/armeabi-v7a.CtsWebkitTestCases:
        • android.webkit.cts.WebSettingsTest.testAccessJavaScriptEnabled
        • android.webkit.cts.WebSettingsTest.testAccessLayoutAlgorithm
    • Android 8.1 – 4.4 + HiKey, 4.14 and X15
      • 4.14.103 / X15 – no regressions
      • 4.4.x / HiKey – no new data
    • Bug Activity
      • 22 – stable WtW
Advertisements

Linaro runs a battery of tests on the Open Embedded and Android operating systems using a variety of hardware and kernel versions in order to detect kernel regressions. These regressions are reported to member companies and the various upstream communities like linux-stable.

This report is a summary of our activity this week.

Testing on Open Embedded

  • KV-126 Final testing for upgrade to sumo happening in staging. Looking to upgrade this week.
  • KV-17 Bisection automation work picking back up
    • Design based on performing an OE build identical to production for a particular board, and then submitting lava job to lkft to determine good/bad
    • Bisection was used last week to find the db410c fix; worked well, needs a lot of cleaning up to become generalized and easy to use.
  • KV-36 It looks like we will be able to add x15s to kernelci without any changes to lkft. Will be pursuing with kernelci.
  • KV-171 “Add LTP tests that android runs to OE/LKFT” finished.
    • Dio tests implemented
    • Commands tests implemented
    • Every test implemented except for ftrace_regression02. We would like to run as many tracing tests as we can, so we split that work out into a new ticket: KV-197 Investigate LTP tracing test cases for LKFT test plan improvement
  • KV-195 Test perf in LKFT based on request from Guenter
    • We used to have a simple perf test; Naresh is going to port it into our environment. User-space tools need to be added to build; kernel config already fine. March/April timeframe.
  • KV-194 Test 64k page size in LKFT request from ARM

Bug Status – 60 open bugs

Linux-Stable LTS RC tested this week

  • 2019-02-25
    • 4.9.161, 4.14.104, 4.19.26, 4.20.13
      • Reported no regressions in <24h
  • 2019-02-21
    • 4.4.176, 4.9.160, 4.14.103, 4.19.25, 4.20.12
      • Reported no regressions in <24h

Testing on Android

  • Discussion
    • Tom followed up with John about hikey kernel configs, He’s optimizing for running on AOSP-master, while LKFT is focused on O-MR1, P, and AOSP-master. The CONFIG_QTAGUID change recently increased failures on P, O given there are test that look for the feature. For kernels on the desserts we’ll need to freeze the config
  • Android 9 / P LTS-premerge – 4.4, 4.9, 4.14, 4.19
    • 4.19.26 / HiKey – No regressions
    • 4.19.25 / HiKey – No regressions
    • 4.14.104 / HiKey – No regressions
    • 4.14.103 / HiKey – No regressions
    • 4.9.161 – Run in progress
    • 4.9.160 / HiKey – No regressions
    • 4.4.176 / HiKey – No regressions
    • Addendum

 

 

    • 4.9.160 / HiKey – failures observed on LKFT but not elsewhere, we have been attempting to reproduce however no failures have been since observed.
      • cts-lkft/armeabi-v7a.CtsLibcoreTestCases/org.apache.harmony.tests.java.net.MulticastSocketTest.test_joinGroupLjava_net_SocketAddressLjava_net_NetworkInterface_IPv4_nullInterface
      • cts-lkft/armeabi-v7a.CtsLibcoreTestCases/org.apache.harmony.tests.java.net.MulticastSocketTest.test_joinGroupLjava_net_SocketAddressLjava_net_NetworkInterface_IPv6
      • cts-lkft/armeabi-v7a.CtsLibcoreTestCases/org.apache.harmony.tests.java.net.MulticastSocketTest.test_joinGroupLjava_net_SocketAddressLjava_net_NetworkInterface_IPv6_nullInterface
      • cts-lkft/armeabi-v7a.CtsLibcoreTestCases/org.apache.harmony.tests.java.net.MulticastSocketTest.test_joinGroupLjava_net_SocketAddressLjava_net_NetworkInterface_multiple_joins_IPv4
      • cts-lkft/armeabi-v7a.CtsLibcoreTestCases/org.apache.harmony.tests.java.net.MulticastSocketTest.test_joinGroupLjava_net_SocketAddressLjava_net_NetworkInterface_multiple_joins_IPv6
      • cts-lkft/armeabi-v7a.CtsLibcoreTestCases/org.apache.harmony.tests.java.net.MulticastSocketTest.test_joinGroup_non_multicast_address_IPv4
      • cts-lkft/armeabi-v7a.CtsLibcoreTestCases/org.apache.harmony.tests.java.net.MulticastSocketTest.test_leaveGroupLjava_net_InetAddress_IPv4
  • Android 9 / P –  4.4, 4.9, 4.14, 4.19 + HiKey
  • AOSP-master-tracking –  4.9, 4.14 4.19 / HiKey & 4.14 / X15
    • We suffered several job failures due to a hub controller issue. The lab has fix. 
      • Example jobs : /usr/local/lab-scripts/cbrxd_hub_control –usb_port 7 –mode sync -i DQ007ADJ failed
    • Regressions found on hi6220-hikey_4.14:
      • cts-lkft-arm64-v8a/arm64-v8a.CtsHardwareTestCases/android.hardware.input.cts.tests.AsusGamepadTestCase.testAllKeys
      • cts-lkft-arm64-v8a/arm64-v8a.CtsHardwareTestCases/android.hardware.input.cts.tests.AsusGamepadTestCase.testAllMotions
      • cts-lkft-arm64-v8a/arm64-v8a.CtsHardwareTestCases/android.hardware.input.cts.tests.SonyDualshock4TestCase.testAllKeys
      • cts-lkft-armeabi-v7a/armeabi-v7a.CtsHardwareTestCases/android.hardware.input.cts.tests.AsusGamepadTestCase.testAllKeys
      • cts-lkft-armeabi-v7a/armeabi-v7a.CtsHardwareTestCases/android.hardware.input.cts.tests.AsusGamepadTestCase.testAllMotions
      • cts-lkft-armeabi-v7a/armeabi-v7a.CtsHardwareTestCases/android.hardware.input.cts.tests.SonyDualshock4TestCase.testAllKeys
    • Regressions found on hi6220-hikey_4.19:
      • cts-lkft-arm64-v8a/arm64-v8a.CtsHardwareTestCases/android.hardware.input.cts.tests.AsusGamepadTestCase.testAllKeys
      • cts-lkft-arm64-v8a/arm64-v8a.CtsHardwareTestCases/android.hardware.input.cts.tests.AsusGamepadTestCase.testAllMotions
      • cts-lkft-armeabi-v7a/armeabi-v7a.CtsHardwareTestCases/android.hardware.input.cts.tests.AsusGamepadTestCase.testAllKeys
      • cts-lkft-armeabi-v7a/armeabi-v7a.CtsHardwareTestCases/android.hardware.input.cts.tests.AsusGamepadTestCase.testAllMotions
    • Regressions found on hi6220-hikey_4.9:
      • cts-lkft-arm64-v8a/arm64-v8a.CtsGraphicsTestCases/android.graphics.cts.VulkanPreTransformTest.testVulkanPreTransformSetToMatchCurrentTransform
      • cts-lkft-arm64-v8a/arm64-v8a.CtsHardwareTestCases/android.hardware.input.cts.tests.AsusGamepadTestCase.testAllKeys
      • cts-lkft-arm64-v8a/arm64-v8a.CtsHardwareTestCases/android.hardware.input.cts.tests.AsusGamepadTestCase.testAllMotions
      • cts-lkft-arm64-v8a/arm64-v8a.CtsHardwareTestCases/android.hardware.input.cts.tests.SonyDualshock4TestCase.testAllKeys
  • Android 8.1 – 4.4 + HiKey, 4.14 and X15
    • 4.14.101 / X15
      • 45 failures – 21 are QTAGUID related –CONFIG_NETFILTER_XT_MATCH_QTAGUID needs to be turned on
    • 4.4.174 / HiKey
      • No regressions!
  • Bugs:
    • 22 – Stable WtW

 

Testing on Open Embedded Linux

Testing on Android

  • Discussion :
    • New combo 4.19 + X15 + P – will be added – anticipate ~ 2 weeks
    • Aosp-master-tracking – kernel version, patchlevel, sublevel will be added qa-reports
    • YongQin – updated LKFT to use latest VTS (R5)
  • Android 9 / P LTS-premerge – 4.4, 4.9, 4.14, 4.19
    • 4.19.23  – no regressions
    • 4.14.101 – no regressions
    • 4.9.158 – no regressions
    • 4.4.174 – no new data
  • Android 9 / P –  4.4, 4.9, 4.14, 4.19 + HiKey
    • 4.19.19 – no regressions
    • 4.14.97 – no regressions
    • 4.9.154 – no regressions
    • 4.4.170 – no new data
  • AOSP-master-tracking –  4.9, 4.14 4.19 / HiKey & 4.14 / X15
  • Android 8.1 – 4.4 + HiKey, 4.14 and X15
    • 4.14.101 / X15
      • 21 regressions – QTAGUID – will work with TI team to adjust their config
    • 4.4.170 / HiKey – no regressions
  • Bugs
    • 22 – Stable WtW
    • 4267, 4268, 4269, 4270 
      • Tom
      • Mykhalio (might have time)
    • 4072 – YongQin
    • 3713 – Sumit

 

The information provided is a wrapup of Linaro’s kernel testing efforts. In particular we are searching for kernel regressions. The report is divided into two parts.

The first involves testing of the Linux kernel using Open Embedded as the user space. Long Term Support kernels (LTS) as well as current stable, mainline and next are uses.

The second part of this report involves testing Linux kernels using Android as the user space. LTS kernels is what are being tested, however these LTS kernels also have the out of tree Android Common patches applied.

Generally but not always new kernel versions are available every week. In the case of testing in Open Embedded, we espeically want to report on RC versions of the LTS kernels within the 48 hour testing window before they are released.

Testing on Open Embedded Linux

Automated reports for kselftest results on -next sending to lkft-triage now

  • Report formatting improved, finished

New work:

  • KV-191 UEFI validation in kernelci
  • Ard Biesheuvel requested support testing UEFI boot mode in kernelci under QEMU. This is looking straight-forward

Bug Status — 59 open bugs

RC Log

2019–02–11

  • 4.9.156, 4.14.99, 4.19.21 — Reported no regressions in <24h
  • 4.20.8 — Reported no regressions in <48h

2019–02–07

  • 4.4.174 — Reported no regressions in <24h

Testing on Android

Discussion :

Android 9 / P LTS-premerge — 4.4, 4.9, 4.14, 4.19

  • 4.19.20 / HiKey — no regressions
  • 4.14.98 / HiKey — no regressions
  • 4.9.155 / HiKey — no regressions
  • 4.4.174 / HiKey — failed to boot — with fix applied, Vts is clean — no regressions and Cts network regressions were still there. The fix as it turns out was the need to be on the latest clang release clang-r349610
  • 4.4.173 / HiKey — 97 cts failures were observed (networking)

Android 9 / P — 4.4, 4.9, 4.14, 4.19 + HiKey

  • 4.4.170 / HiKey — no new data
  • 4.9.154 / HiKey — no regressions
  • 4.14.97 / HiKey — no data received
  • 4.19.19 / HiKey — no data received

AOSP-master-tracking — 4.9, 4.14 4.19 / HiKey & 4.14 / X15

  • A regression was introduced where boot to UI was not successful. Through the course of this week that problem was diagnosed and fixed.
  • Cts CtsLibcoreTestcases — new failure java.lang.IllegalStateException: No SecureRandom implementation was observed. This has introduced approximately ~2600 failures per kernel/board combination.
  • Network tests failed with “Network unreachable” (x15 & HiKey)

Android 8.1–4.4 + HiKey, 4.14 and X15

  • 4.14.94 / X15 — no new data this week
  • 4.4.170 / HiKey — no new data

Bugs

  • No bug discussion this week as YongQin is just back
  • 22 bugs — no change WtW

Plan for the week

  • Examine aosp-master for understand better if we are looking at new test/AOSP changes, infra structure issues, etc

This report is broken up into two parts, OpenEmbedded and Android. These are the two operating systems we are using to run a battery of tests in order to find regressions in the kernel as new patches are added.

The general list of tests that Linaro runs can be found at https://lkft.linaro.org/tests/

When we look for regressions tests often fall into 3 categories of results. Pass, Fail (a regression, exactly what we want to find) and Known Failure where there is a past history with the testcase. Often known failures are flaky testscases that need to be fixed.

OpenEmbedded

  • Automated reports for kselftest results on -next sending to lkft-triage now

The following tests were fixed and removed from our list of known issues

  • open11
  • fcntl36
  • pselect01
  • pselect01_64
  • inotify08
  • bind03

Bug Status — 61 open bugs

RC Log

  • 4.4.173, 4.9.155, 4.14.98, 4.19.20, 4.20.7
  • LTP/fanotify09 confirmed fixed on 4.14.98 due to backport requested in 4.14.97
  • Reported no regressions in <24h

Android

New prebuilt of clang r349610

  • build tested on 4.4.172, 4.9.153, 4.14.96, 4.19.18
  • Boot tested : 4.19.18, 4.14.96, 4.9.153, 4.4.172
  • VTS tested: 4.19.18, 4.14.96, 4.9.153, 4.4.172
  • 4.4.172 — has 1 VtsKernelProcFileApi regression
  • 4.14.96 has 12 kselftest regressions
  • CTS tested: 4.14.96, 4.9.153, 4.4.172 — No regressions

New LTP 20190115 release

  • New VTS 9.0_r5 with latest LPT 20190115 created for testing
  • Initial 4.19.19, 4.14.97, 4.9.154. run has been made, 4.4.172 is in progress at press time

Android 9 / P LTS-premerge — 4.4, 4.9, 4.14, 4.19

  • 4.19.19–1 regression
  • testUsbSerialReadOnDeviceMatches warrents looking into lsusb -v fails?
  • 4.19.20 — in progress
  • 4.14.97 — no regressions
  • 4.14.98 — in progress
  • 4.9.154 — no regressions
  • 4.9.155 — in progress
  • 4.4.173 — in progress

Android 9 / P — 4.4, 4.9, 4.14, 4.19 + HiKey

  • 4.19.16 — current, no new data
  • 4.14.94 — current, no new data
  • 4.9.150 — current, no new data
  • 4.4.170 — current, no new data

AOSP-master-tracking — 4.9, 4.14 4.19 / HiKey & 4.14 / X15

Bugs

  • 22 — Steady WtW

This week the Linux based testing the uses Openembedded upgraded to a newer version of LTP. The Android based testing will make a similar upgrade in another week or two.

Two sets of LTS releases were made during the course of the week this report covers. No regressions were observed on Linux nor with Android.

OpenEmbedded

  • LTP “mm” tests added to LKFT (75 tests/board)
  • Upgraded kselftest that is run against all stable kernels to 4.20.

Bug Status — 65 open bugs

RC Log

2019–01–30

  • 4.9.154, 4.14.97, 4.19.19, 4.20.6
  • LTP upgraded to 20190115 for all branches
  • Reported no regressions in <48h

2019–01–24

  • 4.4.172, 4.9.153, 4.14.96, 4.19.18, 4.20.5
  • kselftest upgraded to 4.20 for all LTS branches
  • Reported no regressions in <24h

Android

Discussion

  • 4.19 has a fairly high number of failures as part of it’s baseline. Started to look into improving baseline
  • 40 VTS failures due to QTAGUID on 4.19, moving to known failures.
  • New LTP testcases, consistent failures

vts-test/arm64-v8a.VtsKernelLtp/VtsKernelLtp.io.aio01_64bit fail

vts-test/arm64-v8a.VtsKernelLtp/VtsKernelLtp.io.aio02_64bit fail

vts-test/arm64-v8a.VtsKernelLtp/VtsKernelLtp.syscalls.io_setup01_64bit fail

vts-test/arm64-v8a.VtsKernelLtp/VtsKernelLtp.syscalls.io_submit01_64bit fail

vts-test/arm64-v8a.VtsKernelLtp/VtsKernelLtp.syscalls.select04_64bit fail

vts-test/armeabi-v7a.VtsKernelLtp/VtsKernelLtp.io.aio01_32bit fail

vts-test/armeabi-v7a.VtsKernelLtp/VtsKernelLtp.io.aio02_32bit fail

vts-test/armeabi-v7a.VtsKernelLtp/VtsKernelLtp.syscalls.io_setup01_32bit fail

vts-test/armeabi-v7a.VtsKernelLtp/VtsKernelLtp.syscalls.io_submit01_32bit fail

Android 9 / P LTS-premerge — 4.4, 4.9, 4.14, 4.19

  • 4.19.18 — no regressions
  • 4.19.17 — no regressions
  • 4.14.96 — no regressions
  • 4.14.95 — no regressions
  • 4.9.153 — no regressions
  • 4.9.152 — no regressions
  • 4.4.172 — no regressions
  • 4.4.171 — no regressions

Android 9 / P — 4.4, 4.9, 4.14, 4.19 + HiKey

  • 4.19.16 — current, rerun completed for missing CTS from last week otherwise no new data
  • 4.14.94 — current, no new data
  • 4.9.150 — current, no new data
  • 4.4.170 — current, no new data

AOSP-master-tracking — 4.9, 4.14 4.19 / HiKey & 4.14 / X15

Android 8.1–4.4 + HiKey, 4.14 and X15

  • 4.14.94 / X15 — seems PVR driver is not loading (outdated). Need to test locally, build and upload the driver.
  • 4.4.170 / HiKey — current, no new data

Bugs

One of the things that we do at Linaro is testing Linux Kernels to look for kernel regressions. Ideally we want a world where those that make use of Long Term Support Kernels (LTS) can depend on the stream of fixes that are being provided.

Mobile phone companies, Linux Distros, embedded Linux deployments, etc all generally like the idea of installing one major version of Linux (e.g. 4.9) and sticking with it for the lifetime of their product.

This, and following stories tell how week to week testing of Linux kernels is going, what we’ve found, or better, not found as the kernel versions tick by.

We test using two host user spaces, open embedded and Android.

Open Embedded

2019–01–21

4.9.152, 4.14.95, 4.20.4

  • Reported crashes in v4.20.3–15-g5592f5bf010b which were intentional ‘canaries’ (the canary successfully died)
  • Reported no regressions in <24h

4.19.17

  • Reported no regressions in <48h

Bug Status — 57 open bugs

Android

Android 9 / P — 4.4, 4.9, 4.14, 4.19 on HiKey

  • 4.14.94 — no regressions
  • 4.19.16 — Note USB OTG regression and potential eMMC issue documented in the bugs section
  • 4.4.170 — no regressions
  • 4.9.150 — no regressions

Android 8.1–4.4 on HiKey, Android 8.1, 4.14 on X15

  • 4.14.94 / X15 no regressions
  • 4.4.170 / HiKey no regressions

Android 9/P + automerged latest version of LTS 4.4, 4.9, 4.14, 4.19 + HiKey + Latest LTP

  • This new combination is a work in progress to pull in latest LTP from AOSP-master, as well as using the combination of Android Common + HiKey Linaro (auto merged). It triggers automatically when Android Common is updated right after a new LTS release is merged. This combo thus gives everyone great visibility to test results nearly immediately after an new LTS is available.
  • We have initial data but are not sharing them as part of this report yet.

AOSP-master tracking with 4.4, 4.9, 4.14, 4.19 on HiKey

  • These builds are being reworked / repackaged so we’ll have data to report next week.

Bugs