Design and memory management of game server engine

C + + programs are sensitive to memory management and often have memory leakage bug s. At the same time, in order to quickly apply and release memory and reduce small memory fragments, most of them have memory management modules.

I added memory management in my own server framework, which is basically the set of STL library. For the memory less than 128 bytes, I manage the application and release in the memory pool, while for the memory application and release of the calling system greater than 128, I just add a memory recording module internally. When the DEBUG flag is turned on, I will record the total size, used size and remaining size of the current memory pool. When you need to detect or find the application record, you can output the current memory situation to the file.

//Here is a brief introduction to the principle and Implementation:

The memory pool manages memory applications of less than 128 bytes. If the memory application is greater than 128 bytes, it will directly call the malloc of the system. If it is less than 128 bytes, it will be applied in the memory pool. The memory pool maintains 16 linked lists. In each linked list, there are memory blocks that have been applied for allocation, and their granularity is 8 bytes. For example, the first linked list is full of 8-byte memory blocks, while the second linked list is full of 16 byte memory blocks, and the third linked list is full of 24 byte memory blocks, and so on until 128 bytes is the 16th linked list. When the business layer wants to apply for memory with a byte size of 1 to 8, it will take out the pre allocated memory nodes in the first linked list and return them to the business layer. To apply for a memory size of 9 to 16, take out a node from the second linked list and allocate it. If the business layer releases memory, it will be inserted into the corresponding linked list according to the corresponding size. If the linked list is empty, allocate N memory blocks of corresponding size from a large memory applied in advance and insert them into the linked list. If the pre applied large memory is also allocated, call the malloc of the system again to apply for a large memory. All memory nodes are allocated on this large memory block. This avoids small pieces of memory fragmentation.

When the DEBUG mode is enabled, the file and line number of the application will be recorded when applying for memory, which is used to troubleshoot problems and count the memory usage of each module.

//Post: memory pool header file code

// 8-byte alignment (granularity)
	#define ALIGN 8

	// Maximum size allocated in pool
	#define MAX_SIZE 128

	// Number of categories in memory pool (MAX_SIZE / ALIGN)
	#define TYPE_COUNT 16

	// 8-byte auxiliary offset
	#define ALIGN_OFFSET 3

	// Memory size requested at one time
	#define ALLOC_MAX_SIZE 65536

	// Number of memory pre allocated at one time
	#define ONE_ALLOC_COUNT 32

	// Memory pool
	class CMemoryPool
	{
	private:
		// Memory node
		struct mem_node_t
		{
			union
			{
				mem_node_t* next;
				void* data;
			};
		};

		// Bucket node
		struct mem_chunk_t
		{
			mem_chunk_t* next;
			size_t size;
		};

		// The memory monitoring node monitors memory usage
		struct mem_check_t
		{
		public:
			mem_check_t(void* p, size_t n, const char* name, int line):
				pMemory(p),nSize(n),nLine(line),fileName(name) {}
			mem_check_t(const mem_check_t& cpy) :
				pMemory(cpy.pMemory), nSize(cpy.nSize), nLine(cpy.nLine),
				fileName(cpy.fileName) {}
			void* pMemory;
			size_t nSize;
			int nLine;
			std::string fileName;
		};

		struct mem_check_t1
		{
			int useSize = 0;
			int allocCount = 0;
			int freeCount = 0;
		};

	public:
		CMemoryPool();
		virtual ~CMemoryPool();

	public:
		// Request memory
		void* Alloc(size_t size);
		
		// Free memory
		void Free(void* pData, size_t size);

		// Request memory
		void* AllocDebug(size_t size, const char* fileName, int nLine);

		// Free memory
		void FreeDebug(void* pData, size_t size);
	public:
		// Set DEBUG mode (enable memory monitoring)
		void SetDebug(bool isDebug);

		// Gets the total size of the memory pool
		int GetMemoryPoolSize();

		// Gets the current remaining size
		int GetMemoryRemianSize();

		// In DEBUG mode, the memory condition is output, and the memory condition is output to the log file
		void SaveDebugDump();

	private:
		// Pre allocated memory
		void refill(size_t size);

		// Memory requested from unallocated memory
		char* chunkalloc(int size, int& count);

		// Implementation of itoa
		char* itoa_(int val, char* buf, int size);
	private:

		// Pre allocated memory pool queue
		mem_node_t* m_pPool[TYPE_COUNT];

		// Unallocated memory block
		mem_chunk_t* m_pChunk;
		char* m_pBegin;
		char* m_pEnd;

		// Is it debug mode
		bool m_bDebug;

		// Memory monitoring list
		std::map<long long, mem_check_t> m_mapCheck;
		std::map<std::string, mem_check_t1>  m_mapCountCheck;
		size_t check_size;
		size_t max_use_size;
		size_t max_use_count;
	};

//When the business logic requests memory, call:

// Request memory
void* CMemoryPool::Alloc(size_t size)
{
	if (size == 0)
	{
		return (void*)"";
	}

    // If it is larger than the maximum memory size managed by the memory pool, directly call the system memory allocation
	if (size > MAX_SIZE)
	{
		void* p = malloc(size);
		if (!p)
		{
			// Failed to request memory
			printf("(CMemoryPool::Alloc)not enough memory, size:%lld", size);
			abort();
			return NULL;
		}

		return p;
	}

    // Locate the corresponding memory list of the memory pool according to the requested memory size
	mem_node_t** freeList = m_pPool + FREE_INDEX(size);
	if (!(*freeList))
	{
        // If there is no memory in the current linked list, apply to the internal pool to expand the amount of memory of this size
		refill(size);
	}

    // Check again for expanded memory
	if (!(*freeList))
	{
		return NULL;
	}

    // Take out the first memory block of this size in the linked list,
    // Return to business caller
	mem_node_t* pNew = (*freeList);
	(*freeList) = (*freeList)->next;
	return (void*)pNew;
}

//First, directly call the system interface if the memory size to be applied is greater than 128:

if (size > MAX_SIZE)
	{
		void* p = malloc(size);
		if (!p)
		{
			// Failed to request memory
			printf("(CMemoryPool::Alloc)no enough memory, size:%lld", size);
			abort();
			return NULL;
		}

		return p;
	}

//Then calculate the index of the linked list for the memory size to be applied:

// Here is the macro definition
#define FREE_INDEX(bytes) ((bytes - size_t(1)) >> (int)ALIGN_OFFSET)
mem_node_t** freeList = m_pPool + FREE_INDEX(size);
	if (!(*freeList))
	{
		refill(size);
	}

//If the current linked list is empty, N memory blocks of corresponding size need to be allocated to the linked list from the large unallocated memory in the memory pool

// Allocate memory
void CMemoryPool::refill(size_t size)
{
	// byte alignment 
	size = (size + ((size_t)ALIGN) - 1) & ~(((size_t)ALIGN) - 1);
	
	// Pre allocated number
	int nodeCount = ONE_ALLOC_COUNT;

	// Request memory
	char* chunk = chunkalloc((int)size, nodeCount);
	if (!chunk)
	{
		return;
	}
	
	mem_node_t** pNodeList = m_pPool + FREE_INDEX(size);
	mem_node_t* curNode;
	for (int i = 0; i < nodeCount; ++i)
	{
		mem_node_t* pNewNodeList = (mem_node_t*)(chunk + (size * i));
		curNode = pNewNodeList;
		curNode->next = *pNodeList;
		*pNodeList = curNode;
	}

	return;
}

//Implementation of chunkalloc:

// Memory never allocated in request
char* CMemoryPool::chunkalloc(int size, int& count)
{
	// Total size of application
	int maxSize = size * count;
	
	// Remaining size
	int remainSize =(int)(m_pEnd - m_pBegin);
	
	// Determine whether the memory is sufficient
	if (remainSize >= maxSize)
	{
		char* p = m_pBegin;
		m_pBegin += maxSize;
		return p;
	}

	// Not enough memory, but check whether there is enough space for at least one
	else if (remainSize >= size)
	{
		count = remainSize / size;
		int allocCount = count * size;
		char* p = m_pBegin;
		m_pBegin += allocCount;
		return p;
	}
	else
	{
		// None. Add the remaining space to the free linked list and re apply for new memory allocation
		if (remainSize > 0)
		{
			mem_node_t** pNodeList = m_pPool + FREE_INDEX(remainSize);
			mem_node_t* pNewFree = (mem_node_t*)m_pBegin;
			pNewFree->next = *pNodeList;
			*pNodeList = pNewFree;
			m_pBegin = m_pEnd = NULL;
		}

		// Re apply for new memory, with a fixed size of 64kb each time. This applies to the total memory pool
		//int newSize = maxSize * 2;
		int newSize = ALLOC_MAX_SIZE;
		char* pBuff = (char*)malloc(newSize);
		if (!pBuff)
		{
			printf("CMemoryPool::chunkalloc faild : no enough memory.\n");
			abort();
			return NULL;
		}
		mem_chunk_t* pChunk = (mem_chunk_t*)pBuff;
		m_pBegin = pBuff + sizeof(mem_chunk_t);
		m_pEnd = m_pBegin + ALLOC_MAX_SIZE - sizeof(mem_chunk_t);
		pChunk->size = ALLOC_MAX_SIZE - sizeof(mem_chunk_t);
		pChunk->next = m_pChunk;
		m_pChunk = pChunk;

		return chunkalloc(size, count);
	}
}

//The above is the explanation of the application memory code, and the following is the implementation of the application function in DEBUG mode, which mainly adds the code segment for recording memory usage:

// Request memory
void* CMemoryPool::AllocDebug(size_t size, const char* fileName, int nLine)
{
    // Follow the normal application process
	void* pData = Alloc(size);
	if (!pData)
	{
		return NULL;
	}

    // DEBUG mode records the file name and line number of each memory application
	if (m_bDebug && fileName && fileName[0] != '\0')
	{
		char* file1 = strrchr(const_cast<char*>(fileName), FILE_MARK);
		char buf[32] = { '\0' };
		std::string name(file1 ? (file1 + 1) : fileName);
		name += ":";
		name += itoa_(nLine, buf, 10);
		max_use_size += (int)size;
		m_mapCheck.insert(std::make_pair((long long)pData, mem_check_t(pData, size, name.c_str(), nLine)));
		auto& p = m_mapCountCheck[name];
		p.allocCount += 1;
		p.useSize += (int)size;
	}

	return pData;
}

//Free memory:

// Free memory
void CMemoryPool::Free(void* pData, size_t size)
{
	if (size == 0 || !pData)
	{
		printf("CMemoryPool::Free faild : size = %lld pData = %lld\n", size, (long long)pData);
		return ;
	}

	if (size > MAX_SIZE)
	{
		free(pData);
		return;
	}

	mem_node_t** freeList = m_pPool + FREE_INDEX(size);
	mem_node_t* pData1 = (mem_node_t*)pData;
	pData1->next = *freeList;
	*freeList = pData1;
	return;
}

//debug mode to free memory: added code snippet to delete application records

// Free memory
void CMemoryPool::FreeDebug(void* pData, size_t size)
{
	Free(pData, size);

	if (!m_bDebug)
	{
		return;
	}
    
    // Delete application record here
	max_use_size -= size;
	std::string fileName = "";
	auto iter = m_mapCheck.find((long long)pData);
	if (iter != m_mapCheck.end())
	{
		fileName = iter->second.fileName;
		m_mapCheck.erase(iter);
		m_mapCountCheck[fileName].freeCount += 1;
	}
}

//Print memory usage in debug mode:

// In DEBUG mode, the memory condition is output, and the memory condition is output to the log file
void CMemoryPool::SaveDebugDump()
{
	std::string fileName = "memory_pool_monitor.log";
	FILE* fp = fopen(fileName.c_str(), "wb");

	if (NULL == fp)
	{
		return;
	}

	fprintf(fp, "memory pool max size : %d, remain size  : %d use size : %lld\r\n\n",
		GetMemoryPoolSize(), GetMemoryRemianSize(), max_use_size);

	fprintf(fp, "===== memory pool use info: ========\r\n\n");
	for (auto iter = m_mapCountCheck.begin(); iter != m_mapCountCheck.end(); ++iter)
	{
		fprintf(fp, "%s -> alloc count : %d, free count : %d, now use size : %d \r\n",
			iter->first.c_str(), iter->second.allocCount, iter->second.freeCount, iter->second.useSize);
	}

	fprintf(fp, "\n======block point count : %lld =========\r\n\n", m_mapCheck.size());
	for (auto iter = m_mapCheck.begin(); iter != m_mapCheck.end(); ++iter)
	{

		fprintf(fp, "block pointer:%p size:%lld file: %s\r\n",
			iter->second.pMemory, iter->second.nSize, iter->second.fileName.c_str());
	}

	fclose(fp);

	return;
}

The implementation of memory pool is basically here. The code encapsulated in memory pool is pasted below:

// Request memory
	extern void* CoreDataAlloc(int size);
	// Free memory
	extern void CoreDataFree(void* ptr, int size);
	// Request memory
	extern void* CoreDataAllocDebug(int size, const char* fileName, int nLine);
	// Free memory
	extern void CoreDataFreeDebug(void* ptr, int size);

	template <typename T>
	T* CoreDataNew(T* p)
	{
		T* ptr = (T*)CoreDataAlloc(sizeof(T));
		if (!ptr)
		{
			return ptr;
		}

		new (ptr) T();

		return ptr;
	}

	template <typename T>
	void CoreDataDelete(T* ptr)
	{
		if (!ptr)
		{
			return;
		}
		ptr->~T();

		CoreDataFree(ptr, sizeof(T));
	}

	template <typename T>
	T* CoreDataNewDebug(T* p, const char* fileName, int nLine)
	{
		T* ptr = (T*)CoreDataAllocDebug(sizeof(T), fileName, nLine);
		if (!ptr)
		{
			return ptr;
		}

		new (ptr) T();

		return ptr;
	}

	template <typename T>
	void CoreDataDeleteDebug(T* ptr)
	{
		if (!ptr)
		{
			return;
		}
		ptr->~T();

		CoreDataFreeDebug(ptr, sizeof(T));
	}

#ifdef _DEBUG
#define CORE_ALLOC(size) CoreData::CoreDataAllocDebug(size, __FILE__, __LINE__)
#define CORE_FREE(ptr, size) CoreData::CoreDataFreeDebug(ptr, size)
#define CORE_NEW(type) CoreData::CoreDataNewDebug((type*)0, __FILE__, __LINE__)
#define CORE_DELETE(ptr) CoreData::CoreDataDeleteDebug(ptr)
#else
#define CORE_ALLOC(size) CoreData::CoreDataAlloc(size)
#define CORE_FREE(ptr, size) CoreData::CoreDataFree(ptr, size)
#define CORE_NEW(type) CoreData::CoreDataNew((type*)0, __FILE__, __LINE__)
#define CORE_DELETE(ptr) CoreData::CoreDataDelete(ptr)
#endif

Now there are many third-party libraries of open source memory management, such as jemalloc. I also use them. You can study them if you have nothing to do

Tags: C++ Back-end server architecture Game Development

Posted by JimChuD on Wed, 13 Apr 2022 02:38:20 +0300