Is it possible to prevent the recognized message from being lost in kafka?
source link: https://www.codesd.com/item/is-it-possible-to-prevent-the-recognized-message-from-being-lost-in-kafka.html
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
Is it possible to prevent the recognized message from being lost in kafka?
We consider using kafka as critical messaging middle-ware.
But it looks like message durability guarantee is optimistic in kafka replication design:
For better performance, each follower sends an acknowledgment after the message is written to memory. So, for each committed message, we guarantee that the message is stored in multiple replicas in memory However, there is no guarantee that any replica has persisted the commit message to disks though.
In worst case, if whole cluster outage at same time before flush acknowledged messages to disk, some data may get lost. Is it possible to avoid this case?
There are multiple configurations to adjust frequency of log flushes. You can increase the time the flush scheduler thread checks if a flush is necessary log.flush.scheduler.interval.ms and you can decrease the number of messages needed to trigger a flush log.flush.interval.messages.
Although you'd never need to worry about this case if you're able to replicate across different data centers.
Recommend
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK