PG14归档失败解决办法archiver failed on wal_lsn

问题描述

昨晚Repmgr+PG14主备主库因wal日志撑爆磁盘，删除主库过期wal文件重做备库后上午进行主备状态巡查，主库向备库发送wal文件正常，但是查主库状态时发现显示有1条归档失败的记录。
postgres: archiver failed on 000000010000006F00000086

  • 主库：

walsender repmgr 172.28.32.23(36122) streaming 72/1BAC3A10" walsender正常
archiver failed on 000000010000006F00000086" 归档失败

  • 备库：

walreceiver streaming 77/9EB6A198" “” “” " walreceiver正常

--查主库数据库状态
[root@pgmaster ~]# systemctl status postgres
● postgres.service - PostgreSQL database server
Loaded: loaded (/usr/lib/systemd/system/postgres.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2023-10-12 22:04:08 CST; 13h ago
Process: 3710968 ExecStart=/server/data/pgdb/pgsql/bin/pg_ctl start -D $PGDATA (code=exited, status=0/SUCCESS)
Main PID: 3710970 (postgres)
Tasks: 53 (limit: 201967)
Memory: 19.0G
CGroup: /system.slice/postgres.service
├─ 3710970 /server/data/pgdb/pgsql/bin/postgres -D /server/data/pgdb/data
├─ 3710971 "postgres: logger " "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
├─ 3710992 "postgres: checkpointer " "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
├─ 3710993 "postgres: background writer " "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
├─ 3710994 "postgres: walwriter " "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
├─ 3710995 "postgres: archiver failed on 000000010000006F00000086" "" "" "" "" "" "" "" "" ""
├─ 3710996 "postgres: logical replication launcher " "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
├─ 3711001 "postgres: top_portal top_portal 172.28.32.18(41438) idle" "" "" "" "" "" ""
├─ 3711003 "postgres: tj_sjjh dataexchange 172.28.32.28(35406) idle" "" "" "" "" "" "" ""
├─ 3711009 "postgres: repmgr repmgr 172.28.32.22(64096) idle" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
├─ 3711468 "postgres: top_portal top_portal 172.28.32.18(41720) idle" "" "" "" "" "" ""
├─ 3713807 "postgres: top_portal top_portal 172.28.32.20(44492) idle" "" "" "" "" "" ""
├─ 3723017 "postgres: walsender repmgr 172.28.32.23(36122) streaming 72/1BAC3A10"  #wal 发送正常--查备库状态
[root@pgslave ~]# systemctl status postgres
● postgres.service - PostgreSQL database server
Loaded: loaded (/usr/lib/systemd/system/postgres.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2023-10-13 00:12:19 CST; 12h ago
Process: 1931221 ExecStart=/server/data/pgdb/pgsql/bin/pg_ctl start -D $PGDATA (code=exited, status=0/SUCCESS)
Main PID: 1931223 (postgres)
Tasks: 7 (limit: 201967)
Memory: 23.2G
CGroup: /system.slice/postgres.service
├─ 1931223 /server/data/pgdb/pgsql/bin/postgres -D /server/data/pgdb/data
├─ 1931224 "postgres: logger " "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
├─ 1931225 "postgres: startup recovering 00000001000000770000009E" "" "" "" "" "" "" "" "" ""
├─ 1931226 "postgres: checkpointer " "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
├─ 1931227 "postgres: background writer " "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
├─ 1931230 "postgres: walreceiver streaming 77/9EB6A198" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""   #wal接收
└─ 1931430 "postgres: repmgr repmgr 172.28.32.23(22956) idle" "" "" "" "" "" "" "" "" "" "" "" "" "" ""Oct 13 00:12:17 pgslave systemd[1]: Starting PostgreSQL database server...
Oct 13 00:12:17 pgslave pg_ctl[1931221]: waiting for server to start....
Oct 13 00:12:17 pgslave pg_ctl[1931223]: 2023-10-13 00:12:17.497 CST [1931223] LOG:  redirecting log output to logging collector process
Oct 13 00:12:17 pgslave pg_ctl[1931223]: 2023-10-13 00:12:17.497 CST [1931223] HINT:  Future log output will appear in directory "log".
Oct 13 00:12:19 pgslave pg_ctl[1931221]: . done
Oct 13 00:12:19 pgslave pg_ctl[1931221]: server started
Oct 13 00:12:19 pgslave systemd[1]: Started PostgreSQL database server.

问题分析

1.查看数据库日志

在这里插入图片描述

2.查看归档配置参数

参数配置正确，归档目录权限也正确

postgres=# show archive_command;archive_command                      
-----------------------------------------------------------/usr/bin/lz4 -q -z %p /server/data/pgdb/pg_archive/%f.lz4
(1 row)postgres=# show archive_mode;archive_mode 
--------------on
(1 row)--查看归档目录的权限
[postgres@pgmaster ~]$ ls -ld /server/data/pgdb/pg_archive
drwxr-x--- 2 postgres postgres 4214784 Oct 13 13:14 /server/data/pgdb/pg_archive

3.手动切日志

手工归档成功，但是未解决，查看状态依然时卡住归档失败的那条wal记录那里

--手工归档
top_portal=# select pg_switch_wal();pg_switch_wal 
---------------72/51C4CFD8
(1 row)--查主库数据库状态
[root@pgmaster ~]# systemctl status postgres
● postgres.service - PostgreSQL database server
Loaded: loaded (/usr/lib/systemd/system/postgres.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2023-10-12 22:04:08 CST; 13h ago
Process: 3710968 ExecStart=/server/data/pgdb/pgsql/bin/pg_ctl start -D $PGDATA (code=exited, status=0/SUCCESS)
Main PID: 3710970 (postgres)
Tasks: 53 (limit: 201967)
Memory: 19.0G
CGroup: /system.slice/postgres.service
├─ 3710970 /server/data/pgdb/pgsql/bin/postgres -D /server/data/pgdb/data
├─ 3710971 "postgres: logger " "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
├─ 3710992 "postgres: checkpointer " "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
├─ 3710993 "postgres: background writer " "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
├─ 3710994 "postgres: walwriter " "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
├─ 3710995 "postgres: archiver failed on 000000010000006F00000086" "" "" "" "" "" "" "" "" ""
├─ 3710996 "postgres: logical replication launcher " "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
├─ 3711001 "postgres: top_portal top_portal 172.28.32.18(41438) idle" "" "" "" "" "" ""
├─ 3711003 "postgres: tj_sjjh dataexchange 172.28.32.28(35406) idle" "" "" "" "" "" "" ""
├─ 3711009 "postgres: repmgr repmgr 172.28.32.22(64096) idle" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
├─ 3711468 "postgres: top_portal top_portal 172.28.32.18(41720) idle" "" "" "" "" "" ""
├─ 3713807 "postgres: top_portal top_portal 172.28.32.20(44492) idle" "" "" "" "" "" ""
├─ 3723017 "postgres: walsender repmgr 172.28.32.23(36122) streaming 72/1BAC3A10"  #wal 发送正常--查当前wal_lsn
top_portal=# select pg_current_wal_lsn();pg_current_wal_lsn 
--------------------72/52638F10
(1 row)--查当前wal_lsn对应的wal文件
top_portal=# select pg_walfile_name(pg_current_wal_lsn());pg_walfile_name      
--------------------------000000010000007200000052
(1 row)--查当前最新检查点,最新检查点之前的wal文件均可以删除
[postgres@pgmaster ~]$ pg_controldata $PGDATA
pg_control version number:            1300
Catalog version number:               202107181
Database system identifier:           7268852449124462799
Database cluster state:               in production
pg_control last modified:             Fri 13 Oct 2023 10:07:35 AM CST  
Latest checkpoint location:           71/CDD2FF28
Latest checkpoint's REDO location:    71/CDD28F18
Latest checkpoint's REDO WAL file:    0000000100000071000000CD--查报错中的wal文件
[postgres@pgmaster pg_wal]$ ls -l 000000010000006F00000086
-rw------- 1 postgres postgres 16777216 Oct 12 21:12 000000010000006F00000086
[postgres@pgmaster pg_wal]$ find /server/data/pgdb/pg_archive -name 000000010000006F00000086*
ls: cannot access '000000010000006F00000086': No such file or directory
[postgres@pgmaster pg_wal]$ find /server -name 000000010000006F00000086*
-rw------- 1 postgres postgres 16777216 Oct 12 21:12 000000010000006F00000086

4.检查$PGDATA/pg_wal/archive_status/目录下文件

[postgres@pgmaster ~]$ cd /server/data/pgdb/data/pg_wal/archive_status/
[postgres@pgmaster archive_status]$ ls -l *.ready
ls: cannot access '*.ready': No such file or directory说明不存在需要归档但没归档的文件

该目录下，ready说明是需要归档但是没归档的，done是归档完成了的

解决办法

1.将归档失败的wal文件备份到/home/postgres目录下(生产环境如果磁盘空间允许切记不要rm删除，mv备份到目标位置)
2.手工归档select pg_switch_wal();
3.再次查看主备库状态

--1.将归档失败的wal文件备份到/home/postgres目录下
[postgres@pgmaster pg_wal]$ mv 000000010000006F00000086 /home/postgres/000000010000006F00000086
[postgres@pgmaster pg_wal]$ ls -l /home/postgres/000000010000006F00000086
-rw------- 1 postgres postgres 16777216 Oct 12 21:12 /home/postgres/000000010000006F00000086--2.手工归档
postgres=# select pg_switch_wal();pg_switch_wal 
---------------73/7EF502E0
(1 row)--3.再次查看主库状态显示正常
[root@pgmaster data]# systemctl status postgres
● postgres.service - PostgreSQL database serverLoaded: loaded (/usr/lib/systemd/system/postgres.service; enabled; vendor preset: disabled)Active: active (running) since Thu 2023-10-12 22:04:08 CST; 13h agoProcess: 3710968 ExecStart=/server/data/pgdb/pgsql/bin/pg_ctl start -D $PGDATA (code=exited, status=0/SUCCESS)Main PID: 3710970 (postgres)Tasks: 50 (limit: 201967)Memory: 26.6GCGroup: /system.slice/postgres.service├─ 3710970 /server/data/pgdb/pgsql/bin/postgres -D /server/data/pgdb/data├─ 3710971 "postgres: logger " "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""├─ 3710992 "postgres: checkpointer " "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""├─ 3710993 "postgres: background writer " "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""├─ 3710994 "postgres: walwriter " "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""├─ 3710995 "postgres: archiver archiving 000000010000007100000035" "" "" "" "" "" "" "" "" ""├─ 3710996 "postgres: logical replication launcher " "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""├─ 3711001 "postgres: top_portal top_portal 172.28.32.18(41438) idle" "" "" "" "" "" ""├─ 3711003 "postgres: tj_sjjh dataexchange 172.28.32.28(35406) idle" "" "" "" "" "" "" ""├─ 3711009 "postgres: repmgr repmgr 172.28.32.22(64096) idle" "" "" "" "" "" "" "" "" "" "" "" "" "" ""├─ 3711468 "postgres: top_portal top_portal 172.28.32.18(41720) idle" "" "" "" "" "" ""├─ 3713807 "postgres: top_portal top_portal 172.28.32.20(44492) idle" "" "" "" "" "" ""├─ 3723017 "postgres: walsender repmgr 172.28.32.23(36122) streaming 73/7F000BD0"

补充：若$PGDATA/pg_wal/archive_status/目录下存在大量的*.ready文件

可能的原因分析：如果数据库是突然断电，那么可能arvchive命令没有完全完成，归档目录会存在不完整的文件名称，重启数据库后，会出现归档失败的情况，这个时候，需要去归档目录删除相关归档失败文件，那么归档就会重新归档。
还未遇到该场景的问题，暂未实验。
参考链接

本文来自互联网用户投稿，该文观点仅代表作者本人，不代表本站立场。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如若转载，请注明出处：http://www.hqwc.cn/news/133377.html

如若内容造成侵权/违法违规/事实不符，请联系编程知识网进行投诉反馈email:809451989@qq.com，一经查实，立即删除！

相关文章

甘特图组件DHTMLX Gantt示例 - 如何有效管理团队工作时间？（一）

甘特图组件DHTMLX Gantt示例 - 如何有效管理团队工作时间？（一）

如果没有有效的时间管理工具，如工作时间日历，很难想象一个项目如何成功运转。这就是为什么我们的开发团队非常重视项目管理，并提供了多种选择来安排DHTMLX Gantt的工作时间。使用DHTMLX Gantt这个JavaScript库，您可以创建一个强大…
阅读更多...
Java架构师缓存架构设计解决方案

Java架构师缓存架构设计解决方案

目录 1 缓存常见的三大问题1.1 缓存雪崩1.2 缓存穿透1.3 缓存击穿2 缓存key的生成策略3 热点数据集中失效的问题4 如何提高缓存的命中率5 缓存和数据库双写不一致的问题6 如何对缓存数据进行分片1 缓存常见的三大问题 缓存雪崩、缓存穿透和缓存击穿是缓存系统中常见的三大问题，…
阅读更多...
日常学习记录随笔-大数据之日志(hadoop)收集实战

日常学习记录随笔-大数据之日志(hadoop)收集实战

数据收集(nginx)--->数据分析---> 数据清洗--->数据聚合计算---数据展示 可能涉及到zabix 做任务调度我们的项目 电商日志分析 比如说我们现在有一个系统,我们的数仓建立也要有一个主题 我这个项目是什么我要干什么定义方向 对用户进行分析,用户信息 要懂整个数据的流…
阅读更多...
理解LoadRunner，基于此工具进行后端性能测试的详细过程（上）

理解LoadRunner，基于此工具进行后端性能测试的详细过程（上）

1、LoadRunner 的基本原理 后端性能测试工具通过虚拟用户脚本生成器生成基于协议的虚拟用户脚本，然后根据性能测试场景设计的要求，通过压力控制器控制协调各个压力产生器以并发的方式执行虚拟用户脚本，并且在测试执行过程中，通过…
阅读更多...
【手写数字识别】数据挖掘实验二

【手写数字识别】数据挖掘实验二

文章目录 Ⅰ、项目任务要求任务描述：主要任务要求(必须完成以下内容但不限于这些内容)： II、实现过程数据集描述实验运行环境描述KNN模型决策树模型朴素贝叶斯模型SVM模型不同方法对MNIST数据集分类识别结果分析(不同方法识别对比率表及结果分析) 完整代…
阅读更多...
2023最新pytorch安装教程，简单易懂，面向初学者（Anaconda+GPU）

2023最新pytorch安装教程，简单易懂，面向初学者（Anaconda+GPU）

一、前言 目前是2023.1.27,鉴于本人安装过程中踩得坑，安装之前我先给即将安装pytorch的各位提个醒，有以下几点需要注意 1.判断自己电脑是否有GPU 注意这点很重要，本教程面向有NVIDA显卡的电脑，如果你的电脑没有GPU或者使用AMD显…
阅读更多...
2023-10-12 LeetCode每日一题（找出数组的串联值）

2023-10-12 LeetCode每日一题（找出数组的串联值）

2023-10-12每日一题 一、题目编号 2562. 找出数组的串联值二、题目链接 点击跳转到题目位置 三、题目描述 给你一个下标从 0 开始的整数数组 nums 。 现定义两个数字的 串联 是由这两个数值串联起来形成的新数字。 例如，15 和 49 的串联是 1549 。 nums 的 串…
阅读更多...
QT运行界面与画布大小不一致问题（一步到位）

QT运行界面与画布大小不一致问题（一步到位）

QT运行界面与画布大小不一致问题 出现的问题直接设置环境变量main函数中输入以下代码更改系统缩放比好了，看一下运行结果吧 出现的问题 当我们运行程序时，发现运行出来的大小和设计的几面大小有很大的差别，这使我们开发起来就特别的困难&…
阅读更多...
iOS 面试题以及自我理解答案

iOS 面试题以及自我理解答案

1、简述push原理，push的证书和其他的有什么不一样？ 第 一阶段：BeejiveIM服务器把要发送的消息、目的iPhone的标识打包，发给APNS。 第二阶段：APNS在自身的已注册Push服务 的iPhone列表中，查找有相应标识的iP…
阅读更多...
数字孪生技术：新零售的未来之路

数字孪生技术：新零售的未来之路

随着科技的不断进步，新零售产业正经历着巨大的变革。数字孪生作为一种新兴技术正在加速这一变革的进程。它不仅为新零售企业带来了更高效的运营方式，还为消费者提供了更个性化、便捷的购物体验。那么，数字孪生技术究竟如何在新零售产业中发挥…
阅读更多...
angular项目指定端口，实现局域网内ip访问

angular项目指定端口，实现局域网内ip访问

直接修改package.json文件 "dev": "ng serve --host 0.0.0.0 --port 8080"终端运行npm run dev启动项目。 这里就指定了使用8080端口运行项目，同时局域网内的其他电脑可以通过访问运行项目主机的ip来访问项目 例如项目运行在ip地址为192.168.2…
阅读更多...
BI工具：让数据分析井然有序一望而知

BI工具：让数据分析井然有序一望而知

BI（Business Intelligence）工具是一类专门用于数据分析和决策支持的软件工具。 它们能够将企业内部和外部的数据进行整合、处理和可视化，帮助用户从海量数据中获取有价值的见解和洞察，并以直观、易懂的方式展示给决策者和相关人员…
阅读更多...
推荐文章
最新文章

Copyright @ 2022~2023