summaryrefslogtreecommitdiff
path: root/include/jemalloc
Commit message (Collapse)AuthorAgeFilesLines
* Allow overriding `LG_PAGE`HEADdevKevin Svetlitski2023-05-172-0/+18
| | | | | This is useful for our internal builds where we override the configuration in the header files generated by autoconf.
* Remove dead stores detected by static analysisKevin Svetlitski2023-05-111-2/+1
| | | | | | | None of these are harmful, and they are almost certainly optimized away by the compiler. The motivation for fixing them anyway is that we'd like to enable static analysis as part of CI, and the first step towards that is resolving the warnings it produces at present.
* Add config detection for JEMALLOC_HAVE_PTHREAD_SET_NAME_NP.Qi Wang2023-05-111-0/+3
| | | | and use it on the background thread name setting.
* Make eligible functions `static`Kevin Svetlitski2023-05-081-2/+0
| | | | | | The codebase is already very disciplined in making any function which can be `static`, but there are a few that appear to have slipped through the cracks.
* Make `edata_cmp_summary_comp` 30% fasterKevin Svetlitski2023-05-041-16/+19
| | | | | | | | | | | | | `edata_cmp_summary_comp` is one of the very hottest functions, taking up 3% of all time spent inside Jemalloc. I noticed that all existing callsites rely only on the sign of the value returned by this function, so I came up with this equivalent branchless implementation which preserves this property. After empirical measurement, I have found that this implementation is 30% faster, therefore representing a 1% speed-up to the allocator as a whole. At @interwq's suggestion, I've applied the same optimization to `edata_esnead_comp` in case this function becomes hotter in the future.
* Some nits in cache_bin.hAmaury Séchet2023-05-011-4/+4
|
* Remove locked flag set in malloc_mutex_trylockguangli-dai2023-04-061-1/+0
| | | | | As a hint flag of the lock, parameter locked should be set only when the lock is gained or freed.
* Disallow decay during reentrancy.Qi Wang2023-04-052-11/+25
| | | | | | Decay should not be triggered during reentrant calls (may cause lock order reversal / deadlocks). Added a delay_trigger flag to the tickers to bypass decay when rentrancy_level is not zero.
* Rearrange the bools in prof_tdata_t to save some bytes.Qi Wang2023-04-051-3/+3
| | | | | | This lowered the sizeof(prof_tdata_t) from 200 to 192 which is a round size class. Afterwards the tdata_t size remain unchanged with the last commit, which effectively inlined the storage of thread names for free.
* Inline the storage for thread name in prof_tdata_t.Qi Wang2023-04-054-4/+40
| | | | | | | | | The previous approach managed the thread name in a separate buffer, which causes races because the thread name update (triggered by new samples) can happen at the same time as prof dumping (which reads the thread names) -- these two operations are under separate locks to avoid blocking each other. Implemented the thread name storage as part of the tdata struct, which resolves the lifetime issue and also avoids internal alloc / dalloc during prof_sample.
* Simplify the logic in ph_removeAmaury Séchet2023-03-311-44/+20
|
* Do not maintain root->prev in ph_remove.Amaury Séchet2023-03-311-3/+0
|
* Simplify the logic in ph_insertAmaury Séchet2023-03-311-29/+30
| | | | | | Also fixes what looks like an off by one error in the lazy aux list merge part of the code that previously never touched the last node in the aux list.
* Fix the rdtscp detection bug and add prefix for the macro.guangli-dai2023-03-232-1/+7
|
* Fix exception specification error for hosts using musl libcMarvin Schmidt2023-03-161-1/+1
| | | | | | | | | | | | | | | | | | | | | | | It turns out that the previous commit did not suffice since the JEMALLOC_SYS_NOTHROW definition also causes the same exception specification errors as JEMALLOC_USE_CXX_THROW did: ``` x86_64-pc-linux-musl-cc -std=gnu11 -Werror=unknown-warning-option -Wall -Wextra -Wshorten-64-to-32 -Wsign-compare -Wundef -Wno-format-zero-length -Wpointer- arith -Wno-missing-braces -Wno-missing-field-initializers -pipe -g3 -fvisibility=hidden -Wimplicit-fallthrough -O3 -funroll-loops -march=native -O2 -pipe -c -march=native -O2 -pipe -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/background_thread.o src/background_thread.c In file included from src/jemalloc_cpp.cpp:9: In file included from include/jemalloc/internal/jemalloc_preamble.h:27: include/jemalloc/internal/../jemalloc.h:254:32: error: exception specification in declaration does not match previous declaration void JEMALLOC_SYS_NOTHROW *je_malloc(size_t size) ^ include/jemalloc/internal/../jemalloc.h:75:21: note: expanded from macro 'je_malloc' ^ /usr/x86_64-pc-linux-musl/include/stdlib.h:40:7: note: previous declaration is here void *malloc (size_t); ^ ``` On systems using the musl C library we have to omit the exception specification on malloc function family like it's done for MacOS, FreeBSD and OpenBSD.
* Remove unused mutex from hpa_centralAmaury Séchet2023-03-101-5/+0
|
* Use asm volatile during benchmarks.guangli-dai2023-02-241-0/+3
|
* Implement prof sample hooks "experimental.hooks.prof_sample(_free)".Qi Wang2022-12-073-3/+17
| | | | | | | | | | | The added hooks hooks.prof_sample and hooks.prof_sample_free are intended to allow advanced users to track additional information, to enable new ways of profiling on top of the jemalloc heap profile and sample features. The sample hook is invoked after the allocation and backtracing, and forwards the both the allocation and backtrace to the user hook; the sample_free hook happens before the actual deallocation, and forwards only the ptr and usz to the hook.
* Inline free and sdallocx into operator deleteGuangli Dai2022-11-213-6/+227
|
* Add a configure option --enable-force-getenv.Qi Wang2022-11-041-0/+6
| | | | | | Allows the use of getenv() rather than secure_getenv() to read MALLOC_CONF. This helps in situations where hosts are under full control, and setting MALLOC_CONF is needed while also setuid. Disabled by default.
* Enable fast thread locals for dealloc-only threads.Qi Wang2022-10-251-0/+4
| | | | | | | | | | Previously if a thread does only allocations, it stays on the slow path / minimal initialized state forever. However, dealloc-only is a valid pattern for dedicated reclamation threads -- this means thread cache is disabled (no batched flush) for them, which causes high overhead and contention. Added the condition to fully initialize TSD when a fair amount of dealloc activities are observed.
* jemalloc_internal_types.h: Use alloca if __STDC_NO_VLA__ is definedPaul Smith2022-10-141-1/+1
| | | | | | | | | No currently-available version of Visual Studio C compiler supports variable length arrays, even if it defines __STDC_VERSION__ >= C99. As far as I know Microsoft has no plans to ever support VLAs in MSVC. The C11 standard requires that the __STDC_NO_VLA__ macro be defined if the compiler doesn't support VLAs, so fall back to alloca() if so.
* Fix safety_check segfault in double free testdivanorama2022-10-031-2/+1
|
* fix build for non linux/BSD platforms.David Carlier2022-10-031-0/+3
|
* Add arena-level name.Guangli Dai2022-09-163-0/+7
| | | | An arena-level name can help identify manual arenas.
* Making jemalloc max stack depth a runtime optionGuangli Dai2022-09-123-2/+8
|
* Refactor the remote / cross thread cache bin stats readingGuangli Dai2022-09-061-50/+50
| | | | Refactored cache_bin.h so that only one function is racy.
* Add double free detection using slab bitmap for debug buildGuangli Dai2022-09-061-16/+55
| | | | Add a sanity check for double free issue in the arena in case that the tcache has been flushed.
* Add double free detection in thread cache for debug buildIvan Zaitsev2022-08-043-0/+37
| | | | | | Add new runtime option `debug_double_free_max_scan` that specifies the max number of stack entries to scan in the cache bit when trying to detect the double free bug (currently debug build only).
* enabled percpu_arena settings on macOs.David CARLIER2022-07-191-1/+1
| | | | follow-up on #2280
* OpenBSD build fix. still no cpu affinity.David Carlier2022-07-192-2/+2
| | | | | - enabling pthread_get/pthread_set_name_np api. - disabling per thread cpu affinity handling, unsupported on this platform.
* New configure option '--enable-pageid' for LinuxDavid Carlier2022-06-091-0/+6
| | | | | | | The option makes jemalloc use prctl with PR_SET_VMA to tag memory mappings with "jemalloc_pg" or "jemalloc_pg_overcommit". This allows to easily identify jemalloc's mappings in /proc/<pid>/maps. PR_SET_VMA is only available in Linux 5.17 and above.
* Implement malloc_getcpu for amd64 and arm64 macOSDavid Carlier2022-06-081-0/+9
| | | | This enables per CPU arena on MacOS
* jemalloc: fix PowerPC definitions in quantum.hbarracuda1562022-05-261-1/+1
|
* Implement pvalloc replacementAlex Lapenkou2022-05-184-0/+9
| | | | | Despite being an obsolete function, pvalloc is still present in GLIBC and should work correctly when jemalloc replaces libc allocator.
* Fix compiling edata.h with MSVCYuriy Chernyshov2022-05-091-2/+4
| | | | | At the time an attempt to compile jemalloc 5.3.0 with MSVC 2019 results in the followin error message: > jemalloc/include/jemalloc/internal/edata.h:660: error C4576: a parenthesized type followed by an initializer list is a non-standard explicit type conversion syntax
* Make the default option of zero realloc match the system allocator.Qi Wang2022-05-051-0/+3
|
* Avoid abort() in test/integration/cpp/infallible_new_true.Qi Wang2022-04-251-1/+4
| | | | | Allow setting the safety check abort hook through mallctl, which avoids abort() and core dumps.
* fix some typoscuishuang2022-04-254-4/+4
| | | | Signed-off-by: cuishuang <imcusg@gmail.com>
* Rename zero_realloc option "strict" to "alloc".Qi Wang2022-04-201-1/+1
| | | | | With realloc(ptr, 0) being UB per C23, the option name "strict" makes less sense now. Rename to "alloc" which describes the behavior.
* Do not disable SEC by default for 64k pages platformsAlex Lapenkou2022-03-241-1/+1
| | | | | | Default SEC max_alloc option value was 32k, disabling SEC for platforms with lg-page=16. This change enables SEC for all platforms, making minimum max_alloc value equal to PAGE.
* Add comments and use meaningful vars in sz_psz2ind.Charles2022-03-242-9/+40
|
* Implement PAGE_FLOOR macroAlex Lapenkou2022-03-221-0/+3
|
* Fix FreeBSD system jemalloc TSD cleanupAlex Lapenkov2022-03-024-4/+7
| | | | | | | | | | | | | | Before this commit, in case FreeBSD libc jemalloc was overridden by another jemalloc, proper thread shutdown callback was involved only for the overriding jemalloc. A call to _malloc_thread_cleanup from libthr would be redirected to user jemalloc, leaving data about dead threads hanging in system jemalloc. This change tackles the issue in two ways. First, for current and old system jemallocs, which we can not modify, the overriding jemalloc would locate and invoke system cleanup routine. For upcoming jemalloc integrations, the cleanup registering function will also be redirected to user jemalloc, which means that system jemalloc's cleanup routine will be registered in user's jemalloc and a single call to _malloc_thread_cleanup will be sufficient to invoke both callbacks.
* Fix failed assertion due to racy memory accessAlex Lapenkou2022-02-171-16/+59
| | | | | | | | | | While calculating the number of stashed pointers, multiple variables potentially modified by a concurrent thread were used for the calculation. This led to some inconsistencies, correctly detected by the assertions. The change eliminates some possible inconsistencies by using unmodified variables and only once a concurrently modified one. The assertions are omitted for the cases where we acknowledge potential inconsistencies too.
* Add prof_leak_error optionyunxu2022-01-211-0/+1
| | | | | | The option makes the process to exit with error code 1 if a memory leak is detected. This is useful for implementing automated tools that rely on leak detection.
* Simplify arena_migrate() to take arena_t* instead of indices.Qi Wang2022-01-112-2/+2
| | | | | This makes debugging slightly easier and avoids the confusion of "should we create new arenas" here.
* FreeBSD also needs the OS-X "don't declare system functions asCraig Leres2022-01-111-1/+1
| | | | nothrow" fix since it also has jemalloc in the base system
* Fix unnecessary returns in san_(un)guard_pages_two_sided.Qi Wang2022-01-041-3/+2
|
* Rename full_position to low_bound in cache_bin.h.Qi Wang2021-12-291-12/+24
|
View Lorry Controller Status