作者
digoal
日期
2021-04-09
标签
PostgreSQL , wal_decode_buffer_size , WAL decoding buffer , 优化
背景
PostgreSQL 14 preview - 支持循环WAL decoding buffer , 优化逻辑decoding性能
buffer大小参数 : wal_decode_buffer_size , 未来支持使用shared memory管理. 大于buffer的recoder使用palloc()开辟其他内存区域存储.
```
Add circular WAL decoding buffer.
author Thomas Munro tmunro@postgresql.org
Thu, 8 Apr 2021 11:03:34 +0000 (23:03 +1200)
committer Thomas Munro tmunro@postgresql.org
Thu, 8 Apr 2021 11:20:42 +0000 (23:20 +1200)
commit f003d9f8721b3249e4aec8a1946034579d40d42c
tree f53cd4a664fc1212156d32cd6f7dd6da507e0602 tree
parent 323cbe7c7ddcf18aaf24b7f6d682a45a61d4e31b commit | diff
Add circular WAL decoding buffer.
Teach xlogreader.c to decode its output into a circular buffer, to
support optimizations based on looking ahead.
-
XLogReadRecord() works as before, consuming records one by one, and
allowing them to be examined via the traditional XLogRecGetXXX()
macros. -
An alternative new interface XLogNextRecord() is added that returns
pointers to DecodedXLogRecord structs that can be examined directly. -
XLogReadAhead() provides a second cursor that lets you see
further ahead, as long as data is available and there is enough space
in the decoding buffer. This returns DecodedXLogRecord pointers to the
caller, but also adds them to a queue of records that will later be
consumed by XLogNextRecord()/XLogReadRecord().
The buffer's size is controlled with wal_decode_buffer_size. The buffer
could potentially be placed into shared memory, for future projects.
Large records that don't fit in the circular buffer are called
"oversized" and allocated separately with palloc().
Discussion: https://postgr.es/m/CA+hUKGJ4VJN8ttxScUFM8dOKX0BrBiboo5uz1cq=AovOddfHpA@mail.gmail.com
```
```
+/
+ * The decoded contents of a record. This occupies a contiguous region of
+ * memory, with main_data and blocks[n].data pointing to memory after the
+ * members declared here.
+ /
+typedef struct DecodedXLogRecord
+{
+ / Private member used for resource management. /
+ size_t size; / total size of decoded record /
+ bool oversized; / outside the regular decode buffer? /
+ struct DecodedXLogRecord next; / decoded record queue link /
+
+ / Public members. /
+ XLogRecPtr lsn; / location /
+ XLogRecPtr next_lsn; / location of next record /
+ XLogRecord header; / header /
+ RepOriginId record_origin;
+ TransactionId toplevel_xid; / XID of top-level transaction /
+ char main_data; / record's main data portion /
+ uint32 main_data_len; / main data portion's length /
+ int max_block_id; / highest block_id in use (-1 if none) /
+ DecodedBkpBlock blocks[FLEXIBLE_ARRAY_MEMBER];
+} DecodedXLogRecord;
+
- /*
-
- Buffer for decoded records. This is a circular buffer, though
-
- individual records can't be split in the middle, so some space is often
-
- wasted at the end. Oversized records that don't fit in this space are
-
- allocated separately.
- */
- char *decode_buffer;
- size_t decode_buffer_size;
- bool free_decode_buffer; / need to free? /
- char decode_buffer_head; / write head */
- char decode_buffer_tail; / read head */
- /*
-
- Queue of records that have been decoded. This is a linked list that
-
- usually consists of consecutive records in decode_buffer, but may also
-
- contain oversized records allocated with palloc().
- */
- DecodedXLogRecord decode_queue_head; / newest decoded record */
- DecodedXLogRecord decode_queue_tail; / oldest decoded record */
- ```
PostgreSQL 许愿链接
您的愿望将传达给PG kernel hacker、数据库厂商等, 帮助提高数据库产品质量和功能, 说不定下一个PG版本就有您提出的功能点. 针对非常好的提议，奖励限量版PG文化衫、纪念品、贴纸、PG热门书籍等，奖品丰富，快来许愿。开不开森.
9.9元购买3个月阿里云RDS PostgreSQL实例
PostgreSQL 解决方案集合
德哥 / digoal's github - 公益是一辈子的事.
