MichaelDBA

blackjack_staging=# select now(), state, query_start, to_char( (round(extract(epoch from (now() - (query_start)))) || ' second')::interval, 'HH24:MI:SS') as duration, xact_start, query from pg_stat_activity where pid = 2349;
              now              | state  |          query_start          | duration |          xact_start           |                                                          query          
-------------------------------+--------+-------------------------------+----------+-------------------------------+-------------------------------------------------------------------------------------------------------------------------
 2016-09-26 09:53:57.912076-05 | active | 2016-09-26 09:43:55.443251-05 | 00:10:02 | 2016-09-26 09:43:55.299271-05 | SELECT count(*) FROM pg_catalog.pg_class AS c WHERE c.oid = to_regclass('"user_capability"') AND c.relpersistence = 'u'
(1 row)

MichaelDBA

Same problem same WAL file:
016-08-16 02:58:28.984 CDT replicator@[unknown][17378:idle] 10.100.128.210(49089) walreceiver : ERROR: requested WAL segment 00000002000022F400000096 has already been removed

The oldest WAL on the master is dated yesterday august 15 at 3:17pm. The backup started yesterday at 8:40pm, over 5 hours after the oldest WAL on the master as shown below:

-rw-r--r-- 1 postgres postgres 33 May 12 2015 00000002.history
-rw------- 1 postgres postgres 16777216 Aug 15 15:17 000000020000230F000000E2
-rw------- 1 postgres postgres 16777216 Aug 15 15:17 000000020000230F000000E3
-rw------- 1 postgres postgres 16777216 Aug 15 15:18 000000020000230F000000E4
-rw------- 1 postgres postgres 16777216 Aug 15 15:18 000000020000230F000000E5