delta/llvm.git/libc/test/src/string, branch main github.com: llvm/llvm-project.git [libc] Add optimized memcpy for RISCV 2023-05-10T08:42:07+00:00 Guillaume Chatelet gchatelet@google.com 2023-05-10T08:34:05+00:00 f4a35492504d7a47afc8ea5b5dd9c437b7b66380 This patch adds two versions of memcpy optimized for architectures where unaligned accesses are either illegal or extremely slow. It is currently enabled for RISCV 64 and RISCV 32 but it could be used for ARM 32 architectures as well. Here is the before / after output of `libc.benchmarks.memory_functions.opt_host --benchmark_filter=BM_Memcpy` on a quad core Linux starfive RISCV 64 board running at 1.5GHz. Before: ``` Run on (4 X 1500 MHz CPU s) CPU Caches: L1 Instruction 32 KiB (x4) L1 Data 32 KiB (x4) L2 Unified 2048 KiB (x1) ------------------------------------------------------------------------ Benchmark Time CPU Iterations UserCounters... ------------------------------------------------------------------------ BM_Memcpy/0/0 474 ns 474 ns 1483776 bytes_per_cycle=0.243492/s bytes_per_second=348.318M/s items_per_second=2.11097M/s __llvm_libc::memcpy,memcpy Google A BM_Memcpy/1/0 210 ns 209 ns 3649536 bytes_per_cycle=0.233819/s bytes_per_second=334.481M/s items_per_second=4.77519M/s __llvm_libc::memcpy,memcpy Google B BM_Memcpy/2/0 1814 ns 1814 ns 396288 bytes_per_cycle=0.247899/s bytes_per_second=354.622M/s items_per_second=551.402k/s __llvm_libc::memcpy,memcpy Google D BM_Memcpy/3/0 89.3 ns 89.2 ns 7459840 bytes_per_cycle=0.217415/s bytes_per_second=311.014M/s items_per_second=11.2071M/s __llvm_libc::memcpy,memcpy Google L BM_Memcpy/4/0 134 ns 134 ns 3815424 bytes_per_cycle=0.226584/s bytes_per_second=324.131M/s items_per_second=7.44567M/s __llvm_libc::memcpy,memcpy Google M BM_Memcpy/5/0 52.8 ns 52.6 ns 11001856 bytes_per_cycle=0.194893/s bytes_per_second=278.797M/s items_per_second=19.0284M/s __llvm_libc::memcpy,memcpy Google Q BM_Memcpy/6/0 180 ns 180 ns 4101120 bytes_per_cycle=0.231884/s bytes_per_second=331.713M/s items_per_second=5.55957M/s __llvm_libc::memcpy,memcpy Google S BM_Memcpy/7/0 195 ns 195 ns 3906560 bytes_per_cycle=0.232951/s bytes_per_second=333.239M/s items_per_second=5.1217M/s __llvm_libc::memcpy,memcpy Google U BM_Memcpy/8/0 152 ns 152 ns 4789248 bytes_per_cycle=0.227507/s bytes_per_second=325.452M/s items_per_second=6.58187M/s __llvm_libc::memcpy,memcpy Google W BM_Memcpy/9/0 6036 ns 6033 ns 118784 bytes_per_cycle=0.249158/s bytes_per_second=356.423M/s items_per_second=165.75k/s __llvm_libc::memcpy,uniform 384 to 4096 ``` After: ``` BM_Memcpy/0/0 126 ns 126 ns 5770240 bytes_per_cycle=1.04707/s bytes_per_second=1.46273G/s items_per_second=7.9385M/s __llvm_libc::memcpy,memcpy Google A BM_Memcpy/1/0 75.1 ns 75.0 ns 10204160 bytes_per_cycle=0.691143/s bytes_per_second=988.687M/s items_per_second=13.3289M/s __llvm_libc::memcpy,memcpy Google B BM_Memcpy/2/0 333 ns 333 ns 2174976 bytes_per_cycle=1.39297/s bytes_per_second=1.94596G/s items_per_second=3.00002M/s __llvm_libc::memcpy,memcpy Google D BM_Memcpy/3/0 49.6 ns 49.5 ns 16092160 bytes_per_cycle=0.710161/s bytes_per_second=1015.89M/s items_per_second=20.1844M/s __llvm_libc::memcpy,memcpy Google L BM_Memcpy/4/0 57.7 ns 57.7 ns 11213824 bytes_per_cycle=0.561557/s bytes_per_second=803.314M/s items_per_second=17.3228M/s __llvm_libc::memcpy,memcpy Google M BM_Memcpy/5/0 48.0 ns 47.9 ns 16437248 bytes_per_cycle=0.346708/s bytes_per_second=495.97M/s items_per_second=20.8571M/s __llvm_libc::memcpy,memcpy Google Q BM_Memcpy/6/0 67.5 ns 67.5 ns 10616832 bytes_per_cycle=0.614173/s bytes_per_second=878.582M/s items_per_second=14.8142M/s __llvm_libc::memcpy,memcpy Google S BM_Memcpy/7/0 84.7 ns 84.6 ns 10480640 bytes_per_cycle=0.819077/s bytes_per_second=1.14424G/s items_per_second=11.8174M/s __llvm_libc::memcpy,memcpy Google U BM_Memcpy/8/0 61.7 ns 61.6 ns 11191296 bytes_per_cycle=0.550078/s bytes_per_second=786.893M/s items_per_second=16.2279M/s __llvm_libc::memcpy,memcpy Google W BM_Memcpy/9/0 981 ns 981 ns 703488 bytes_per_cycle=1.52333/s bytes_per_second=2.12807G/s items_per_second=1019.81k/s __llvm_libc::memcpy,uniform 384 to 4096 ``` It is not as good as glibc for now so there's room for improvement. I suspect a path pumping 16 bytes at once given the doubled numbers for large copies. ``` BM_Memcpy/0/1 146 ns 82.5 ns 8576000 bytes_per_cycle=1.35236/s bytes_per_second=1.88922G/s items_per_second=12.1169M/s glibc memcpy,memcpy Google A BM_Memcpy/1/1 112 ns 63.7 ns 10634240 bytes_per_cycle=0.628018/s bytes_per_second=898.387M/s items_per_second=15.702M/s glibc memcpy,memcpy Google B BM_Memcpy/2/1 315 ns 180 ns 4079616 bytes_per_cycle=2.65229/s bytes_per_second=3.7052G/s items_per_second=5.54764M/s glibc memcpy,memcpy Google D BM_Memcpy/3/1 85.3 ns 43.1 ns 15854592 bytes_per_cycle=0.774164/s bytes_per_second=1107.45M/s items_per_second=23.2249M/s glibc memcpy,memcpy Google L BM_Memcpy/4/1 105 ns 54.3 ns 13427712 bytes_per_cycle=0.7793/s bytes_per_second=1114.8M/s items_per_second=18.4109M/s glibc memcpy,memcpy Google M BM_Memcpy/5/1 77.1 ns 43.2 ns 16476160 bytes_per_cycle=0.279808/s bytes_per_second=400.269M/s items_per_second=23.1428M/s glibc memcpy,memcpy Google Q BM_Memcpy/6/1 112 ns 62.7 ns 11236352 bytes_per_cycle=0.676078/s bytes_per_second=967.137M/s items_per_second=15.9387M/s glibc memcpy,memcpy Google S BM_Memcpy/7/1 131 ns 65.5 ns 11751424 bytes_per_cycle=0.965616/s bytes_per_second=1.34895G/s items_per_second=15.2762M/s glibc memcpy,memcpy Google U BM_Memcpy/8/1 104 ns 55.0 ns 12314624 bytes_per_cycle=0.583336/s bytes_per_second=834.468M/s items_per_second=18.1937M/s glibc memcpy,memcpy Google W BM_Memcpy/9/1 932 ns 466 ns 1480704 bytes_per_cycle=3.17342/s bytes_per_second=4.43321G/s items_per_second=2.14679M/s glibc memcpy,uniform 384 to 4096 ``` Reviewed By: sivachandra Differential Revision: https://reviews.llvm.org/D150202 
This patch adds two versions of memcpy optimized for architectures where unaligned accesses are either illegal or extremely slow.
It is currently enabled for RISCV 64 and RISCV 32 but it could be used for ARM 32 architectures as well.

Here is the before / after output of `libc.benchmarks.memory_functions.opt_host --benchmark_filter=BM_Memcpy` on a quad core Linux starfive RISCV 64 board running at 1.5GHz.

Before:
```
Run on (4 X 1500 MHz CPU s)
CPU Caches:
  L1 Instruction 32 KiB (x4)
  L1 Data 32 KiB (x4)
  L2 Unified 2048 KiB (x1)
------------------------------------------------------------------------
Benchmark              Time             CPU   Iterations UserCounters...
------------------------------------------------------------------------
BM_Memcpy/0/0        474 ns          474 ns      1483776 bytes_per_cycle=0.243492/s bytes_per_second=348.318M/s items_per_second=2.11097M/s __llvm_libc::memcpy,memcpy Google A
BM_Memcpy/1/0        210 ns          209 ns      3649536 bytes_per_cycle=0.233819/s bytes_per_second=334.481M/s items_per_second=4.77519M/s __llvm_libc::memcpy,memcpy Google B
BM_Memcpy/2/0       1814 ns         1814 ns       396288 bytes_per_cycle=0.247899/s bytes_per_second=354.622M/s items_per_second=551.402k/s __llvm_libc::memcpy,memcpy Google D
BM_Memcpy/3/0       89.3 ns         89.2 ns      7459840 bytes_per_cycle=0.217415/s bytes_per_second=311.014M/s items_per_second=11.2071M/s __llvm_libc::memcpy,memcpy Google L
BM_Memcpy/4/0        134 ns          134 ns      3815424 bytes_per_cycle=0.226584/s bytes_per_second=324.131M/s items_per_second=7.44567M/s __llvm_libc::memcpy,memcpy Google M
BM_Memcpy/5/0       52.8 ns         52.6 ns     11001856 bytes_per_cycle=0.194893/s bytes_per_second=278.797M/s items_per_second=19.0284M/s __llvm_libc::memcpy,memcpy Google Q
BM_Memcpy/6/0        180 ns          180 ns      4101120 bytes_per_cycle=0.231884/s bytes_per_second=331.713M/s items_per_second=5.55957M/s __llvm_libc::memcpy,memcpy Google S
BM_Memcpy/7/0        195 ns          195 ns      3906560 bytes_per_cycle=0.232951/s bytes_per_second=333.239M/s items_per_second=5.1217M/s __llvm_libc::memcpy,memcpy Google U
BM_Memcpy/8/0        152 ns          152 ns      4789248 bytes_per_cycle=0.227507/s bytes_per_second=325.452M/s items_per_second=6.58187M/s __llvm_libc::memcpy,memcpy Google W
BM_Memcpy/9/0       6036 ns         6033 ns       118784 bytes_per_cycle=0.249158/s bytes_per_second=356.423M/s items_per_second=165.75k/s __llvm_libc::memcpy,uniform 384 to 4096
```

After:
```
BM_Memcpy/0/0        126 ns          126 ns      5770240 bytes_per_cycle=1.04707/s bytes_per_second=1.46273G/s items_per_second=7.9385M/s __llvm_libc::memcpy,memcpy Google A
BM_Memcpy/1/0       75.1 ns         75.0 ns     10204160 bytes_per_cycle=0.691143/s bytes_per_second=988.687M/s items_per_second=13.3289M/s __llvm_libc::memcpy,memcpy Google B
BM_Memcpy/2/0        333 ns          333 ns      2174976 bytes_per_cycle=1.39297/s bytes_per_second=1.94596G/s items_per_second=3.00002M/s __llvm_libc::memcpy,memcpy Google D
BM_Memcpy/3/0       49.6 ns         49.5 ns     16092160 bytes_per_cycle=0.710161/s bytes_per_second=1015.89M/s items_per_second=20.1844M/s __llvm_libc::memcpy,memcpy Google L
BM_Memcpy/4/0       57.7 ns         57.7 ns     11213824 bytes_per_cycle=0.561557/s bytes_per_second=803.314M/s items_per_second=17.3228M/s __llvm_libc::memcpy,memcpy Google M
BM_Memcpy/5/0       48.0 ns         47.9 ns     16437248 bytes_per_cycle=0.346708/s bytes_per_second=495.97M/s items_per_second=20.8571M/s __llvm_libc::memcpy,memcpy Google Q
BM_Memcpy/6/0       67.5 ns         67.5 ns     10616832 bytes_per_cycle=0.614173/s bytes_per_second=878.582M/s items_per_second=14.8142M/s __llvm_libc::memcpy,memcpy Google S
BM_Memcpy/7/0       84.7 ns         84.6 ns     10480640 bytes_per_cycle=0.819077/s bytes_per_second=1.14424G/s items_per_second=11.8174M/s __llvm_libc::memcpy,memcpy Google U
BM_Memcpy/8/0       61.7 ns         61.6 ns     11191296 bytes_per_cycle=0.550078/s bytes_per_second=786.893M/s items_per_second=16.2279M/s __llvm_libc::memcpy,memcpy Google W
BM_Memcpy/9/0        981 ns          981 ns       703488 bytes_per_cycle=1.52333/s bytes_per_second=2.12807G/s items_per_second=1019.81k/s __llvm_libc::memcpy,uniform 384 to 4096
```

It is not as good as glibc for now so there's room for improvement. I suspect a path pumping 16 bytes at once given the doubled numbers for large copies.
```
BM_Memcpy/0/1        146 ns         82.5 ns      8576000 bytes_per_cycle=1.35236/s bytes_per_second=1.88922G/s items_per_second=12.1169M/s glibc memcpy,memcpy Google A
BM_Memcpy/1/1        112 ns         63.7 ns     10634240 bytes_per_cycle=0.628018/s bytes_per_second=898.387M/s items_per_second=15.702M/s glibc memcpy,memcpy Google B
BM_Memcpy/2/1        315 ns          180 ns      4079616 bytes_per_cycle=2.65229/s bytes_per_second=3.7052G/s items_per_second=5.54764M/s glibc memcpy,memcpy Google D
BM_Memcpy/3/1       85.3 ns         43.1 ns     15854592 bytes_per_cycle=0.774164/s bytes_per_second=1107.45M/s items_per_second=23.2249M/s glibc memcpy,memcpy Google L
BM_Memcpy/4/1        105 ns         54.3 ns     13427712 bytes_per_cycle=0.7793/s bytes_per_second=1114.8M/s items_per_second=18.4109M/s glibc memcpy,memcpy Google M
BM_Memcpy/5/1       77.1 ns         43.2 ns     16476160 bytes_per_cycle=0.279808/s bytes_per_second=400.269M/s items_per_second=23.1428M/s glibc memcpy,memcpy Google Q
BM_Memcpy/6/1        112 ns         62.7 ns     11236352 bytes_per_cycle=0.676078/s bytes_per_second=967.137M/s items_per_second=15.9387M/s glibc memcpy,memcpy Google S
BM_Memcpy/7/1        131 ns         65.5 ns     11751424 bytes_per_cycle=0.965616/s bytes_per_second=1.34895G/s items_per_second=15.2762M/s glibc memcpy,memcpy Google U
BM_Memcpy/8/1        104 ns         55.0 ns     12314624 bytes_per_cycle=0.583336/s bytes_per_second=834.468M/s items_per_second=18.1937M/s glibc memcpy,memcpy Google W
BM_Memcpy/9/1        932 ns          466 ns      1480704 bytes_per_cycle=3.17342/s bytes_per_second=4.43321G/s items_per_second=2.14679M/s glibc memcpy,uniform 384 to 4096
```

Reviewed By: sivachandra

Differential Revision: https://reviews.llvm.org/D150202
[libc] Enable running libc unit tests on AMDGPU 2023-05-04T11:32:52+00:00 Joseph Huber jhuber6@vols.utk.edu 2023-05-03T14:19:11+00:00 632fa3798cc3cdb9d91867c1ae978809ec4bb253 The previous patches added the necessary support for global constructors used to register tests. This patch enables the AMDGPU target to build and run the unit tests on the GPU. Currently this only tests the `ctype` tests, but adding more should be straightforward from here on. Reviewed By: sivachandra Differential Revision: https://reviews.llvm.org/D149517 
The previous patches added the necessary support for global constructors
used to register tests. This patch enables the AMDGPU target to build
and run the unit tests on the GPU. Currently this only tests the `ctype`
tests, but adding more should be straightforward from here on.

Reviewed By: sivachandra

Differential Revision: https://reviews.llvm.org/D149517
[libc] Add 'UNIT_TEST_ONLY' and 'HERMETIC_TEST_ONLY' to 'add_libc_test' 2023-05-02T23:51:12+00:00 Joseph Huber jhuber6@vols.utk.edu 2023-05-02T20:54:29+00:00 644b63bd31f55aaaf1ebafc7df7cd7cea2aa26a8 The `add_libc_test` option allows us to enable both kinds of tests in a single option. However, some tests cannot be made hermetic with the current approach. Such as tests that rely on system utilities or libraries. This patch adds two options `UNIT_TEST_ONLY` and `HERMETIC_TEST_ONLY` to offer more fine-grained control over which version gets built. This makes it explicit which version a test supports and why. Depends on D149662 Reviewed By: sivachandra Differential Revision: https://reviews.llvm.org/D149691 
The `add_libc_test` option allows us to enable both kinds of tests in a
single option. However, some tests cannot be made hermetic with the
current approach. Such as tests that rely on system utilities or
libraries. This patch adds two options `UNIT_TEST_ONLY` and
`HERMETIC_TEST_ONLY` to offer more fine-grained control over which
version gets built. This makes it explicit which version a test supports
and why.

Depends on D149662

Reviewed By: sivachandra

Differential Revision: https://reviews.llvm.org/D149691
[libc] Enable the 'string' unit tests to be hermetic 2023-05-02T23:51:11+00:00 Joseph Huber jhuber6@vols.utk.edu 2023-05-02T16:58:47+00:00 9a20e024a25ce1e20556dc4e567a9d943bff4c7e This changes over the string unit tests to the new `add_libc_test` framework. The one test that wasn't changes was the memory utils because it overran the static buffer used for the hermetic test's bump pointer. Depends on D149656 Reviewed By: sivachandra Differential Revision: https://reviews.llvm.org/D149662 
This changes over the string unit tests to the new `add_libc_test`
framework. The one test that wasn't changes was the memory utils because
it overran the static buffer used for the hermetic test's bump pointer.

Depends on D149656

Reviewed By: sivachandra

Differential Revision: https://reviews.llvm.org/D149662
[libc] Remove use of 'assert' from the string tests 2023-05-02T23:51:10+00:00 Joseph Huber jhuber6@vols.utk.edu 2023-05-02T15:29:13+00:00 a417ee9ccfbc59ac8910e77d3138985240d78e28 Currently we use `assert` as a stand-in for the typical test assertions. This is because these functions exist outside of the base test class so we can't use the typical assertion methods. The presence of these asserts makes it difficult to compile these tests in a standalone format. This patch removes all occurrences. Reviewed By: sivachandra Differential Revision: https://reviews.llvm.org/D149656 
Currently we use `assert` as a stand-in for the typical test assertions.
This is because these functions exist outside of the base test class so
we can't use the typical assertion methods. The presence of these
asserts makes it difficult to compile these tests in a standalone
format. This patch removes all occurrences.

Reviewed By: sivachandra

Differential Revision: https://reviews.llvm.org/D149656
[libc] Run all unit tests, irrespective of whether they belong to a test suite. 2023-04-24T22:37:58+00:00 Siva Chandra Reddy sivachandra@google.com 2023-04-20T07:18:43+00:00 e831f73ac034f3a43aa9dcccf7ceb7b269089ef6 Previously, only those unit tests which belonged to a suite were run as part of libc-unit-tests. It meant that unit tests not part of any suite were not being tested. This change makes all unit tests run as part of libc-unit-tests. The convenience function to add a libc unit test suite has been removed and add_custom_target is used in its place. One of the bit-rotting test has along been fixed. Math exhaustive and differential tests are skipped under full build. Reviewed By: michaelrj Differential Revision: https://reviews.llvm.org/D148784 
Previously, only those unit tests which belonged to a suite were run as
part of libc-unit-tests. It meant that unit tests not part of any suite
were not being tested. This change makes all unit tests run as part of
libc-unit-tests. The convenience function to add a libc unit test suite
has been removed and add_custom_target is used in its place. One of the
bit-rotting test has along been fixed. Math exhaustive and differential
tests are skipped under full build.

Reviewed By: michaelrj

Differential Revision: https://reviews.llvm.org/D148784
[reland][libc] handle memset sequence as a separate struct 2023-04-19T08:37:49+00:00 Guillaume Chatelet gchatelet@google.com 2023-04-18T15:16:20+00:00 458734b8b9ab457e3c88b76e87761a8f4c27a290 These sequence of calls don't really make sense for head_tail and loop_and_tail. 
These sequence of calls don't really make sense for head_tail and loop_and_tail.
Revert "[libc] handle memset sequence as a separate struct" 2023-04-18T15:24:24+00:00 Guillaume Chatelet gchatelet@google.com 2023-04-18T15:24:24+00:00 3950e44eb0ad8d12705d9211c76d622c76f387e9 This patch broke the build bots. This reverts commit 6fdc244e9f13d2ce20a26d05513f22a1e099f90a. 
This patch broke the build bots.
This reverts commit 6fdc244e9f13d2ce20a26d05513f22a1e099f90a.
[libc] handle memset sequence as a separate struct 2023-04-18T15:16:32+00:00 Guillaume Chatelet gchatelet@google.com 2023-04-18T15:16:20+00:00 6fdc244e9f13d2ce20a26d05513f22a1e099f90a These sequence of calls don't really make sense for head_tail and loop_and_tail. 
These sequence of calls don't really make sense for head_tail and loop_and_tail.
[libc] Add debug messages to bcmp and memcmp tests 2023-04-18T13:35:34+00:00 Guillaume Chatelet gchatelet@google.com 2023-04-18T13:35:18+00:00 b3c696fb86f9553e67f63f512adc771d11664669