This seems to be a typical problem when HANA DB ran out of memory.
GLOBAL_ALLOCATION_LIMIT (GAL) = 117.34gb (126002196480b), SHARED_MEMORY = 3.22gb (3466637312b), CODE_SIZE = 1.88gb (2023874560b)
PID=3033 (hdbnameserver), PAL=117.47gb (126136414208b), AB=1.78gb (1912262656b), UA=0, U=899.38mb (943077527b), FSL=0b, FMC=0b
PID=76013 (hdbcompileserve), PAL=117.47gb (126136414208b), AB=1.05gb (1132728320b), UA=0, U=241.02mb (252736591b), FSL=0b, FMC=0b, Blocked=1.33gb (1433931776b)
PID=3234 (hdbpreprocessor), PAL=117.47gb (126136414208b), AB=1.15gb (1236631552b), UA=0, U=284.39mb (298213262b), FSL=0b, FMC=0b
PID=3272 (hdbxsengine), PAL=117.47gb (126136414208b), AB=3.19gb (3428057088b), UA=0, U=2.16gb (2322866680b), FSL=0b, FMC=0b
PID=3270 (hdbindexserver), PAL=117.47gb (126136414208b), AB=103.60gb (111244210176b), UA=0, U=87.16gb (93594690511b), FSL=0b, FMC=0b
PID=4363 (hdbwebdispatche), PAL=117.47gb (126136414208b), AB=1.29gb (1390809088b), UA=0, U=463.50mb (486024301b), FSL=0b, FMC=0b
Total allocated memory= 117.19gb (125835210752b)
Total used memory = 96.28gb (103388120744b)
Sum AB = 120344698880
Sum Used = 97897608872
Heap memory fragmentation: 17% (this value may be high if defragmentation does not help solving the current memory request)
Top allocators (component, name, size). Ordered descending by inclusive_size_in_use.
1: System: / 2.16gb (2322866680b)
2: System: Pool 1.70gb (1828285003b)
3: System: Pool/malloc 621.83mb (652043576b)
4: System: Pool/malloc/libhdbexecutor.so 481.55mb (504948256b)
5: Monitoring & Statistical Data: Pool/Statistics 314.64mb (329926040b)
6: System: Pool/ReserveAllocatorBase 288.54mb (302564096b)
7: System: VirtualAlloc 278.37mb (291897344b)
8: System: Pool/RowEngine 192.46mb (201814552b)
9: Statement Execution & Intermediate Results: StackAllocator 101.88mb (106831872b)
10: System: Pool/ReserveAllocatorBase/ReserveForUndoAndCleanupExec 96.14mb (100810752b)
11: System: Pool/ReserveAllocatorBase/ReserveForOnlineCleanup 96.14mb (100810752b)
12: Monitoring & Statistical Data: Pool/RowEngine/MonitorView 92.30mb (96794088b)
13: System: AllocateOnlyAllocator-unlimited 91.06mb (95486160b)
14: Monitoring & Statistical Data: Pool/RowEngine/MonitorView/Resident 87.89mb (92161872b)
15: Monitoring & Statistical Data: Pool/RowEngine/MonitorView/Resident/M_DEV_TRANSACTIONS_HISTORY 87.73mb (92000016b)
16: System: AllocateOnlyAllocator-unlimited/FLA-UL<3145728,1>/MemoryMapLevel2Blocks 87mb (91226112b)
17: System: AllocateOnlyAllocator-unlimited/FLA-UL<3145728,1> 87mb (91226112b)
18: System: Pool/malloc/libhdbrskernel.so 61.64mb (64640360b)
19: System: Pool/malloc/libhdbbasement.so 54.50mb (57151184b)
20: Monitoring & Statistical Data: Pool/M_DELTA_MERGE_STATISTICS 48.82mb (51200512b)
Top allocators (component, name, size). Ordered descending by exclusive_size_in_use.
1: System: Pool/malloc/libhdbexecutor.so 481.55mb (504948256b)
2: Monitoring & Statistical Data: Pool/Statistics 314.64mb (329926040b)
3: System: VirtualAlloc 278.37mb (291897344b)
4: Statement Execution & Intermediate Results: StackAllocator 101.88mb (106831872b)
5: System: Pool/ReserveAllocatorBase/ReserveForUndoAndCleanupExec 96.14mb (100810752b)
6: System: Pool/ReserveAllocatorBase/ReserveForOnlineCleanup 96.14mb (100810752b)
7: Monitoring & Statistical Data: Pool/RowEngine/MonitorView/Resident/M_DEV_TRANSACTIONS_HISTORY 87.73mb (92000016b)
8: System: AllocateOnlyAllocator-unlimited/FLA-UL<3145728,1>/MemoryMapLevel2Blocks 87mb (91226112b)
9: System: Pool/malloc/libhdbrskernel.so 61.64mb (64640360b)
10: System: Pool/malloc/libhdbbasement.so 54.50mb (57151184b)
11: Monitoring & Statistical Data: Pool/M_DELTA_MERGE_STATISTICS 48.82mb (51200512b)
12: System: Pool/ReserveAllocatorBase/ReserveNestedForUndoAndCleanupFile 48.12mb (50466816b)
13: System: Pool/ReserveAllocatorBase/ReserveMainForUndoAndCleanupFile 48.12mb (50466816b)
14: System: Pool/RowEngine/ContainerManager 44.75mb (46923792b)
15: System: Pool/ResourceContainer 33.06mb (34672328b)
16: System: Pool/ExpressionGlobal/Expression 31.25mb (32770936b)
17: System: Pool/FileAllocator/FileFactory[/usr/sap/HDB/SYS/global/hdb/log/mnt00001/hdb00002/] 30mb (31462096b)
18: System: Pool/Metadata/MetadataCache/MetadataGlobalCacheMap 29.52mb (30963760b)
19: Monitoring & Statistical Data: Pool/M_EXECUTED_STATEMENTS 25.63mb (26880896b)
20: Caches: Pool/RowEngine/SQLPlan 17mb (17825864b)
Callstack of failing 'provideMemory':
1: 0x00007f3755147af0 in MemoryManager::SynchronizedMemoryConsumptionInfos::update(ltt::allocator_statistics&, char const*, unsigned long)+0xc0 at LockedScope.hpp:107 (libhdbbasis.so)
2: 0x00007f375514b51a in MemoryManager::GlobalMemoryHandler::provideMemoryAndReturnSize(MemoryManager::Precharge&, bool, bool, unsigned long, unsigned long, bool, MemoryManager::AllocatorConfigurationSPI&, bool, char const*, unsigned long, MemoryManager::TPALPrecharge&)+0xe36 at GlobalMemoryHandler.cpp:4828 (libhdbbasis.so)
3: 0x00007f375514bff3 in MemoryManager::GlobalMemoryHandler::provideMemory(MemoryManager::Precharge&, unsigned long, unsigned long, bool, MemoryManager::AllocatorConfigurationSPI&, bool, char const*, unsigned long)+0x90 at GlobalMemoryHandler.cpp:4493 (libhdbbasis.so)
4: 0x00007f37550e8e80 in MemoryManager::AllocatorConfigurationSPI::provideMemory(MemoryManager::Precharge&, unsigned long, unsigned long, bool, char const*, unsigned long)+0x30 at AllocatorConfigurationImpl.cpp:2858 (libhdbbasis.so)
5: 0x00007f3755170c38 in MemoryManager::MemoryPool::reserveMemoryAndAllocateBigBlock(unsigned long, bool, unsigned short, ltt::allocator_statistics::SubStats&, void const*, bool, Synchronization::LockHandle<Synchronization::Mutex, false>&)+0xe4 at MemoryPool.cpp:2301 (libhdbbasis.so)
6: 0x00007f3755174895 in MemoryManager::MemoryPool::allocateBigOrHugeBlock(unsigned long, unsigned short, ltt::allocator_statistics&, void const*, bool, bool, Synchronization::LockHandle<Synchronization::Mutex, false>&, bool)+0x2a1 at MemoryPool.cpp:1682 (libhdbbasis.so)
7: 0x00007f3755174cc7 in MemoryManager::MemoryPool::allocate(unsigned long, unsigned short, ltt::allocator_statistics&, bool&, bool, bool, void const*)+0x363 at MemoryPool.cpp:1627 (libhdbbasis.so)
8: 0x00007f37551bbbf1 in MemoryManager::PoolAllocator::allocateNoThrowImpl(unsigned long, void const*)+0x80 at PoolAllocator.cpp:1849 (libhdbbasis.so)
9: 0x00007f3756a38ef6 in ltt::allocator::allocate(unsigned long)+0x22 at memory.cpp:83 (libhdbbasement.so)
10: 0x00007f3756381f9f in TrexThreads::ThreadManager::getStatistics(ltt_adp::vector<TrexThreads::ThreadInfo, ltt::integral_constant<bool, true> >*, bool, bool, bool, ltt_adp::map<unsigned long, ltt_adp::vector<unsigned long, ltt::integral_constant<bool, true> >, ltt::less<unsigned long>, ltt::integral_constant<bool, true> >*)+0x144b at memory.hpp:887 (libhdbbasement.so)
11: 0x00007f37601f9b6d in ptime::traceThreadLockInfo()+0x469 at ServiceThreadSamplesMonitor.cc:119 (libhdbrskernel.so)
12: 0x00007f37601fc165 in ptime::ServiceThreadSamplerThread::run(void*)+0x11 at ServiceThreadSamplesMonitor.cc:81 (libhdbrskernel.so)
13: 0x00007f3756377f1a in TrexThreads::PoolThread::run()+0x836 at PoolThread.cpp:389 (libhdbbasement.so)
14: 0x00007f37563799a0 in TrexThreads::PoolThread::run(void*&)+0x10 at PoolThread.cpp:165 (libhdbbasement.so)
15: 0x00007f3754ffa337 in Execution::Thread::staticMainImp(void**)+0x733 at Thread.cpp:462 (libhdbbasis.so)
16: 0x00007f3754ffb928 in Execution::Thread::staticMain(void*)+0x34 at ThreadMain.cpp:26 (libhdbbasis.so)
Top UnifiedTable allocators:
Top allocators (component, name, size). Ordered descending by inclusive_size_in_use.
1: Column Store Tables: UnifiedTableComposite 0b
Top allocators (component, name, size). Ordered descending by exclusive_size_in_use.
1: Column Store Tables: UnifiedTableComposite 0b
Restarted HDB which should have resolved this problem.
This seems to be a typical problem when HANA DB ran out of memory.
![5769f4c8a39c1.jpg](serve/attachment&path=5769f4c8a39c1.jpg)
````
GLOBAL_ALLOCATION_LIMIT (GAL) = 117.34gb (126002196480b), SHARED_MEMORY = 3.22gb (3466637312b), CODE_SIZE = 1.88gb (2023874560b)
PID=3033 (hdbnameserver), PAL=117.47gb (126136414208b), AB=1.78gb (1912262656b), UA=0, U=899.38mb (943077527b), FSL=0b, FMC=0b
PID=76013 (hdbcompileserve), PAL=117.47gb (126136414208b), AB=1.05gb (1132728320b), UA=0, U=241.02mb (252736591b), FSL=0b, FMC=0b, Blocked=1.33gb (1433931776b)
PID=3234 (hdbpreprocessor), PAL=117.47gb (126136414208b), AB=1.15gb (1236631552b), UA=0, U=284.39mb (298213262b), FSL=0b, FMC=0b
PID=3272 (hdbxsengine), PAL=117.47gb (126136414208b), AB=3.19gb (3428057088b), UA=0, U=2.16gb (2322866680b), FSL=0b, FMC=0b
PID=3270 (hdbindexserver), PAL=117.47gb (126136414208b), AB=103.60gb (111244210176b), UA=0, U=87.16gb (93594690511b), FSL=0b, FMC=0b
PID=4363 (hdbwebdispatche), PAL=117.47gb (126136414208b), AB=1.29gb (1390809088b), UA=0, U=463.50mb (486024301b), FSL=0b, FMC=0b
Total allocated memory= 117.19gb (125835210752b)
Total used memory = 96.28gb (103388120744b)
Sum AB = 120344698880
Sum Used = 97897608872
Heap memory fragmentation: 17% (this value may be high if defragmentation does not help solving the current memory request)
Top allocators (component, name, size). Ordered descending by inclusive_size_in_use.
1: System: / 2.16gb (2322866680b)
2: System: Pool 1.70gb (1828285003b)
3: System: Pool/malloc 621.83mb (652043576b)
4: System: Pool/malloc/libhdbexecutor.so 481.55mb (504948256b)
5: Monitoring & Statistical Data: Pool/Statistics 314.64mb (329926040b)
6: System: Pool/ReserveAllocatorBase 288.54mb (302564096b)
7: System: VirtualAlloc 278.37mb (291897344b)
8: System: Pool/RowEngine 192.46mb (201814552b)
9: Statement Execution & Intermediate Results: StackAllocator 101.88mb (106831872b)
10: System: Pool/ReserveAllocatorBase/ReserveForUndoAndCleanupExec 96.14mb (100810752b)
11: System: Pool/ReserveAllocatorBase/ReserveForOnlineCleanup 96.14mb (100810752b)
12: Monitoring & Statistical Data: Pool/RowEngine/MonitorView 92.30mb (96794088b)
13: System: AllocateOnlyAllocator-unlimited 91.06mb (95486160b)
14: Monitoring & Statistical Data: Pool/RowEngine/MonitorView/Resident 87.89mb (92161872b)
15: Monitoring & Statistical Data: Pool/RowEngine/MonitorView/Resident/M_DEV_TRANSACTIONS_HISTORY 87.73mb (92000016b)
16: System: AllocateOnlyAllocator-unlimited/FLA-UL<3145728,1>/MemoryMapLevel2Blocks 87mb (91226112b)
17: System: AllocateOnlyAllocator-unlimited/FLA-UL<3145728,1> 87mb (91226112b)
18: System: Pool/malloc/libhdbrskernel.so 61.64mb (64640360b)
19: System: Pool/malloc/libhdbbasement.so 54.50mb (57151184b)
20: Monitoring & Statistical Data: Pool/M_DELTA_MERGE_STATISTICS 48.82mb (51200512b)
Top allocators (component, name, size). Ordered descending by exclusive_size_in_use.
1: System: Pool/malloc/libhdbexecutor.so 481.55mb (504948256b)
2: Monitoring & Statistical Data: Pool/Statistics 314.64mb (329926040b)
3: System: VirtualAlloc 278.37mb (291897344b)
4: Statement Execution & Intermediate Results: StackAllocator 101.88mb (106831872b)
5: System: Pool/ReserveAllocatorBase/ReserveForUndoAndCleanupExec 96.14mb (100810752b)
6: System: Pool/ReserveAllocatorBase/ReserveForOnlineCleanup 96.14mb (100810752b)
7: Monitoring & Statistical Data: Pool/RowEngine/MonitorView/Resident/M_DEV_TRANSACTIONS_HISTORY 87.73mb (92000016b)
8: System: AllocateOnlyAllocator-unlimited/FLA-UL<3145728,1>/MemoryMapLevel2Blocks 87mb (91226112b)
9: System: Pool/malloc/libhdbrskernel.so 61.64mb (64640360b)
10: System: Pool/malloc/libhdbbasement.so 54.50mb (57151184b)
11: Monitoring & Statistical Data: Pool/M_DELTA_MERGE_STATISTICS 48.82mb (51200512b)
12: System: Pool/ReserveAllocatorBase/ReserveNestedForUndoAndCleanupFile 48.12mb (50466816b)
13: System: Pool/ReserveAllocatorBase/ReserveMainForUndoAndCleanupFile 48.12mb (50466816b)
14: System: Pool/RowEngine/ContainerManager 44.75mb (46923792b)
15: System: Pool/ResourceContainer 33.06mb (34672328b)
16: System: Pool/ExpressionGlobal/Expression 31.25mb (32770936b)
17: System: Pool/FileAllocator/FileFactory[/usr/sap/HDB/SYS/global/hdb/log/mnt00001/hdb00002/] 30mb (31462096b)
18: System: Pool/Metadata/MetadataCache/MetadataGlobalCacheMap 29.52mb (30963760b)
19: Monitoring & Statistical Data: Pool/M_EXECUTED_STATEMENTS 25.63mb (26880896b)
20: Caches: Pool/RowEngine/SQLPlan 17mb (17825864b)
Callstack of failing 'provideMemory':
1: 0x00007f3755147af0 in MemoryManager::SynchronizedMemoryConsumptionInfos::update(ltt::allocator_statistics&, char const*, unsigned long)+0xc0 at LockedScope.hpp:107 (libhdbbasis.so)
2: 0x00007f375514b51a in MemoryManager::GlobalMemoryHandler::provideMemoryAndReturnSize(MemoryManager::Precharge&, bool, bool, unsigned long, unsigned long, bool, MemoryManager::AllocatorConfigurationSPI&, bool, char const*, unsigned long, MemoryManager::TPALPrecharge&)+0xe36 at GlobalMemoryHandler.cpp:4828 (libhdbbasis.so)
3: 0x00007f375514bff3 in MemoryManager::GlobalMemoryHandler::provideMemory(MemoryManager::Precharge&, unsigned long, unsigned long, bool, MemoryManager::AllocatorConfigurationSPI&, bool, char const*, unsigned long)+0x90 at GlobalMemoryHandler.cpp:4493 (libhdbbasis.so)
4: 0x00007f37550e8e80 in MemoryManager::AllocatorConfigurationSPI::provideMemory(MemoryManager::Precharge&, unsigned long, unsigned long, bool, char const*, unsigned long)+0x30 at AllocatorConfigurationImpl.cpp:2858 (libhdbbasis.so)
5: 0x00007f3755170c38 in MemoryManager::MemoryPool::reserveMemoryAndAllocateBigBlock(unsigned long, bool, unsigned short, ltt::allocator_statistics::SubStats&, void const*, bool, Synchronization::LockHandle<Synchronization::Mutex, false>&)+0xe4 at MemoryPool.cpp:2301 (libhdbbasis.so)
6: 0x00007f3755174895 in MemoryManager::MemoryPool::allocateBigOrHugeBlock(unsigned long, unsigned short, ltt::allocator_statistics&, void const*, bool, bool, Synchronization::LockHandle<Synchronization::Mutex, false>&, bool)+0x2a1 at MemoryPool.cpp:1682 (libhdbbasis.so)
7: 0x00007f3755174cc7 in MemoryManager::MemoryPool::allocate(unsigned long, unsigned short, ltt::allocator_statistics&, bool&, bool, bool, void const*)+0x363 at MemoryPool.cpp:1627 (libhdbbasis.so)
8: 0x00007f37551bbbf1 in MemoryManager::PoolAllocator::allocateNoThrowImpl(unsigned long, void const*)+0x80 at PoolAllocator.cpp:1849 (libhdbbasis.so)
9: 0x00007f3756a38ef6 in ltt::allocator::allocate(unsigned long)+0x22 at memory.cpp:83 (libhdbbasement.so)
10: 0x00007f3756381f9f in TrexThreads::ThreadManager::getStatistics(ltt_adp::vector<TrexThreads::ThreadInfo, ltt::integral_constant<bool, true> >*, bool, bool, bool, ltt_adp::map<unsigned long, ltt_adp::vector<unsigned long, ltt::integral_constant<bool, true> >, ltt::less<unsigned long>, ltt::integral_constant<bool, true> >*)+0x144b at memory.hpp:887 (libhdbbasement.so)
11: 0x00007f37601f9b6d in ptime::traceThreadLockInfo()+0x469 at ServiceThreadSamplesMonitor.cc:119 (libhdbrskernel.so)
12: 0x00007f37601fc165 in ptime::ServiceThreadSamplerThread::run(void*)+0x11 at ServiceThreadSamplesMonitor.cc:81 (libhdbrskernel.so)
13: 0x00007f3756377f1a in TrexThreads::PoolThread::run()+0x836 at PoolThread.cpp:389 (libhdbbasement.so)
14: 0x00007f37563799a0 in TrexThreads::PoolThread::run(void*&)+0x10 at PoolThread.cpp:165 (libhdbbasement.so)
15: 0x00007f3754ffa337 in Execution::Thread::staticMainImp(void**)+0x733 at Thread.cpp:462 (libhdbbasis.so)
16: 0x00007f3754ffb928 in Execution::Thread::staticMain(void*)+0x34 at ThreadMain.cpp:26 (libhdbbasis.so)
Top UnifiedTable allocators:
Top allocators (component, name, size). Ordered descending by inclusive_size_in_use.
1: Column Store Tables: UnifiedTableComposite 0b
Top allocators (component, name, size). Ordered descending by exclusive_size_in_use.
1: Column Store Tables: UnifiedTableComposite 0b
````
Restarted HDB which should have resolved this problem.