作者
digoal
日期
2021-05-12
标签
PostgreSQL , streaming logical replication
背景
Generate WAL invalidations message during command completion when using logical replication (Dilip Kumar, Tomas Vondra, Amit Kapila) When logical replication is disabled, WAL invalidation messages are generated at transaction completion. This allows logical streaming of in-progress transactions.
Allow logical decoding to more efficiently process cache invalidation messages (Dilip Kumar) This allows Logical decoding to work efficiently in presence of a large amount of DDL.
Allow control over whether logical decoding messages are sent to the replication stream (David Pirotte, Euler Taveira)
在发布这个patch前, 长事务需要在发送端(pub端)解析, 并存储在logical_decoding_work_mem中, 如果超出则写入disk. 直到事务结束才能将解析的内容发送到订阅端.
这样会有一个问题, 如果是个大事务或者很长的大事务, 接收延迟比较高, 同时在pub端需要占据大量的内存或disk空间来存储未结束的事务decode结果的内容.
PostgreSQL 14 支持logical replication stream模式, 可以解决以上问题, 进行中的事务解析后可以流式发送给订阅端, 不需要hold在pub端. 创建pub的时候设置为streaming=true即可支持.
Allow logical replication to stream long in-progress transactions to standbys (Tomas Vondra, Dilip Kumar, Amit Kapila, Ajin Cherian, Nikhil Sontakke, Stas Kelvich)
Previously transactions that exceeded logical_decoding_work_mem were written to disk until the transaction completed.
https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=7259736a6e5b7c7588fff9578370736a6648acbb
```
Implement streaming mode in ReorderBuffer.
author Amit Kapila akapila@postgresql.org
Sat, 8 Aug 2020 02:04:39 +0000 (07:34 +0530)
committer Amit Kapila akapila@postgresql.org
Sat, 8 Aug 2020 02:17:06 +0000 (07:47 +0530)
commit 7259736a6e5b7c7588fff9578370736a6648acbb
tree a2261d4ed09124a00d9ed8c0082f22256364aa77 tree
parent 0a7d771f0f63eb120e7f0a60aecd543ab25ba197 commit | diff
Implement streaming mode in ReorderBuffer.
Instead of serializing the transaction to disk after reaching the
logical_decoding_work_mem limit in memory, we consume the changes we have
in memory and invoke stream API methods added by commit 45fdc9738b.
However, sometimes if we have incomplete toast or speculative insert we
spill to the disk because we can't generate the complete tuple and stream.
And, as soon as we get the complete tuple we stream the transaction
including the serialized changes.
We can do this incremental processing thanks to having assignments
(associating subxact with toplevel xacts) in WAL right away, and
thanks to logging the invalidation messages at each command end. These
features are added by commits 0bead9af48 and c55040ccd0 respectively.
Now that we can stream in-progress transactions, the concurrent aborts
may cause failures when the output plugin consults catalogs (both system
and user-defined).
We handle such failures by returning ERRCODE_TRANSACTION_ROLLBACK
sqlerrcode from system table scan APIs to the backend or WALSender
decoding a specific uncommitted transaction. The decoding logic on the
receipt of such a sqlerrcode aborts the decoding of the current
transaction and continue with the decoding of other transactions.
We have ReorderBufferTXN pointer in each ReorderBufferChange by which we
know which xact it belongs to. The output plugin can use this to decide
which changes to discard in case of stream_abort_cb (e.g. when a subxact
gets discarded).
We also provide a new option via SQL APIs to fetch the changes being
streamed.
Author: Dilip Kumar, Tomas Vondra, Amit Kapila, Nikhil Sontakke
Reviewed-by: Amit Kapila, Kuntal Ghosh, Ajin Cherian
Tested-by: Neha Sharma, Mahendra Singh Thalor and Ajin Cherian
Discussion: https://postgr.es/m/688b0b7f-2f6c-d827-c27b-216a8e3ea700@2ndquadrant.com
```
https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=464824323e57dc4b397e8b05854d779908b55304
```
Add support for streaming to built-in logical replication.
author Amit Kapila akapila@postgresql.org
Thu, 3 Sep 2020 02:24:07 +0000 (07:54 +0530)
committer Amit Kapila akapila@postgresql.org
Thu, 3 Sep 2020 02:24:07 +0000 (07:54 +0530)
commit 464824323e57dc4b397e8b05854d779908b55304
tree 30a02506ae6b53475302980bc558e2a41ea429f0 tree
parent 66f163068030b5c5fe792a0daee27822dac43791 commit | diff
Add support for streaming to built-in logical replication.
To add support for streaming of in-progress transactions into the
built-in logical replication, we need to do three things:
-
Extend the logical replication protocol, so identify in-progress
transactions, and allow adding additional bits of information (e.g.
XID of subtransactions). -
Modify the output plugin (pgoutput) to implement the new stream
API callbacks, by leveraging the extended replication protocol. -
Modify the replication apply worker, to properly handle streamed
in-progress transaction by spilling the data to disk and then
replaying them on commit.
We however must explicitly disable streaming replication during
replication slot creation, even if the plugin supports it. We
don't need to replicate the changes accumulated during this phase,
and moreover we don't have a replication connection open so we
don't have where to send the data anyway.
Author: Tomas Vondra, Dilip Kumar and Amit Kapila
Reviewed-by: Amit Kapila, Kuntal Ghosh and Ajin Cherian
Tested-by: Neha Sharma, Mahendra Singh Thalor and Ajin Cherian
Discussion: https://postgr.es/m/688b0b7f-2f6c-d827-c27b-216a8e3ea700@2ndquadrant.com
```
https://git.postgresql.org/gitweb/?p=postgresql.git;a=blobdiff;f=doc/src/sgml/ref/create_subscription.sgml;h=b7d7457d004e983cec9346c5f9c6c21d21f5c84b;hp=cdb22c54feabd41e6e48c80fde07cd3133306f57;hb=464824323e57dc4b397e8b05854d779908b55304;hpb=66f163068030b5c5fe792a0daee27822dac43791
+ <varlistentry>
+ <term><literal>streaming</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether streaming of in-progress transactions should
+ be enabled for this subscription. By default, all transactions
+ are fully decoded on the publisher, and only then sent to the
+ subscriber as a whole.
+ </para>
+ </listitem>
+ </varlistentry>
PostgreSQL 许愿链接
您的愿望将传达给PG kernel hacker、数据库厂商等, 帮助提高数据库产品质量和功能, 说不定下一个PG版本就有您提出的功能点. 针对非常好的提议，奖励限量版PG文化衫、纪念品、贴纸、PG热门书籍等，奖品丰富，快来许愿。开不开森.
9.9元购买3个月阿里云RDS PostgreSQL实例
PostgreSQL 解决方案集合
德哥 / digoal's github - 公益是一辈子的事.
