Ryfujk

Do I need to convey a moral for each of my blog post?

Help with identifying unique aircraft over NE Pennsylvania

How old is Nick Fury?

Hot air balloons as primitive bombers

Why are there no stars visible in cislunar space?

Animating wave motion in water

Why do I have a large white artefact on the rendered image?

Can other pieces capture a threatening piece and prevent a checkmate?

Would this string work as string?

When did hardware antialiasing start being available?

Knife as defense against stray dogs

Did Nintendo change its mind about 68000 SNES?

Imaginary part of expression too difficult to calculate

What kind of footwear is suitable for walking in micro gravity environment?

Are hand made posters acceptable in Academia?

Have the tides ever turned twice on any open problem?

Homology of the fiber

How can a new country break out from a developed country without war?

Emojional cryptic crossword

Does convergence of polynomials imply that of its coefficients?

Can a university suspend a student even when he has left university?

"Marked down as someone wanting to sell shares." What does that mean?

Exit shell with shortcut (not typing exit) that closes session properly

Could any one tell what PN is this Chip? Thanks~

Can i delete oplog local.* of Mongo backup fsyncLock() + LVM snapshot?

Sync LVM snapshots to backup serverDoes rsync delete local files if they were deleted on source folder?MongoDB Disaster Prep on AWSLVM+MongoDB. Dropping in speed of backup creationHow to change the master in a MongoDB cluster without restarting the process?Backing up data stored on Amazon S3Huge oplog size after full sync on one secondary causing performance issues - MongoDBDebian safe backup of server system drive?centos dedicated server backup strategy for mongodb and elasticsearch dbMongoDB secondary replica permanently in not reachable/healthy state

1

Is it safe to delete oplog local.* files from a MongoDB snapshot ? Does the fsyncLock() commit oplog entries in DB files ?

I currently do a fsyncLock() + LVM snapshot of Mongo data. Then i mount the snapshot, and remove the oplog local.* files before archiving the files to S3 or FTP (with Duplicity). I do that to reduce the size of archives to transfer to S3.

When i restore on a node on an empty replicaset, oplog files are recreated, and other nodes full resync from this new primary. But i am not sure if there could be missing data in the restore, due to it being in the previously deleted oplog and not written in DB files.

backup mongodb

share|improve this question

asked Dec 27 '14 at 11:06

Alex FAlex F

399515

bumped to the homepage by Community♦ 34 secs ago

This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.

add a comment | 

1

Is it safe to delete oplog local.* files from a MongoDB snapshot ? Does the fsyncLock() commit oplog entries in DB files ?

I currently do a fsyncLock() + LVM snapshot of Mongo data. Then i mount the snapshot, and remove the oplog local.* files before archiving the files to S3 or FTP (with Duplicity). I do that to reduce the size of archives to transfer to S3.

When i restore on a node on an empty replicaset, oplog files are recreated, and other nodes full resync from this new primary. But i am not sure if there could be missing data in the restore, due to it being in the previously deleted oplog and not written in DB files.

backup mongodb

share|improve this question

asked Dec 27 '14 at 11:06

Alex FAlex F

399515

bumped to the homepage by Community♦ 34 secs ago

This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.

add a comment | 

Is it safe to delete oplog local.* files from a MongoDB snapshot ? Does the fsyncLock() commit oplog entries in DB files ?

I currently do a fsyncLock() + LVM snapshot of Mongo data. Then i mount the snapshot, and remove the oplog local.* files before archiving the files to S3 or FTP (with Duplicity). I do that to reduce the size of archives to transfer to S3.

When i restore on a node on an empty replicaset, oplog files are recreated, and other nodes full resync from this new primary. But i am not sure if there could be missing data in the restore, due to it being in the previously deleted oplog and not written in DB files.

backup mongodb

share|improve this question

asked Dec 27 '14 at 11:06

Alex FAlex F

399515

share|improve this question

asked Dec 27 '14 at 11:06

Alex FAlex F

399515

share|improve this question

asked Dec 27 '14 at 11:06

Alex FAlex F

399515

asked Dec 27 '14 at 11:06

Alex FAlex F

399515

asked Dec 27 '14 at 11:06

Alex FAlex F

399515

1 Answer
1

active

oldest

votes

0

Using snapshot functionality for mongo replication back up is not good idea while you run it. First of all you need to get the snapshot of "primary" in your replica set. but the problem is your clients can connect with a wrong write concern policy. like not asking for it or not for majority that leaves uncommitted data in your primary's oplog. even if you keep oplog or all files in local db, you may have problems when you restore (got that once, my primary was looking a rollback with no reason).

my suggestion (i assume you can not stop mongod) is using mongodump, yes it sounds stupid when you can have a snapshot. but the real point of using backups is able to restore them.

if you can't use mongodump or don't want to. Try MMS's cloud backup feature. which can restore whole replica set easily.

Edit:
Another option is running a delayed replica member. here is documentation. which covers most of the backup needs. just determine the delay time. by this way you don't eat up your server's io with running backup/archive tools.

share|improve this answer

edited Dec 28 '14 at 21:13

answered Dec 28 '14 at 0:55

risyasinrisyasin

1,378815

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  The fact is that mongo dump takes more than 24h to restore. The data size is around 300GB. Tt takes so long, that even if it needs to recreate the indexes, it does not make sense, the restore server is not running at full capacity.
  
  – Alex F
  Dec 28 '14 at 21:07
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  and about MMS, we tried, and it was great. Unfortunately, their pricing is completely inadequate for us: we do constant updates. the data size grows slowly, but the oplog operations are numerous, which is why their pricing per oplog operation is killing.
  
  – Alex F
  Dec 28 '14 at 21:12
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  well if you have hundreds of gigabytes, it changes a lot. so i have updated my answer.
  
  – risyasin
  Dec 28 '14 at 21:14
  

  
  
  
  

add a comment | 

Your Answer

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "2"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fserverfault.com%2fquestions%2f654970%2fcan-i-delete-oplog-local-of-mongo-backup-fsynclock-lvm-snapshot%23new-answer', 'question_page');
}
);

Post as a guest

Name

Email

Required, but never shown

0

Using snapshot functionality for mongo replication back up is not good idea while you run it. First of all you need to get the snapshot of "primary" in your replica set. but the problem is your clients can connect with a wrong write concern policy. like not asking for it or not for majority that leaves uncommitted data in your primary's oplog. even if you keep oplog or all files in local db, you may have problems when you restore (got that once, my primary was looking a rollback with no reason).

my suggestion (i assume you can not stop mongod) is using mongodump, yes it sounds stupid when you can have a snapshot. but the real point of using backups is able to restore them.

if you can't use mongodump or don't want to. Try MMS's cloud backup feature. which can restore whole replica set easily.

Edit:
Another option is running a delayed replica member. here is documentation. which covers most of the backup needs. just determine the delay time. by this way you don't eat up your server's io with running backup/archive tools.

share|improve this answer

edited Dec 28 '14 at 21:13

answered Dec 28 '14 at 0:55

risyasinrisyasin

1,378815

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  The fact is that mongo dump takes more than 24h to restore. The data size is around 300GB. Tt takes so long, that even if it needs to recreate the indexes, it does not make sense, the restore server is not running at full capacity.
  
  – Alex F
  Dec 28 '14 at 21:07
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  and about MMS, we tried, and it was great. Unfortunately, their pricing is completely inadequate for us: we do constant updates. the data size grows slowly, but the oplog operations are numerous, which is why their pricing per oplog operation is killing.
  
  – Alex F
  Dec 28 '14 at 21:12
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  well if you have hundreds of gigabytes, it changes a lot. so i have updated my answer.
  
  – risyasin
  Dec 28 '14 at 21:14
  

  
  
  
  

add a comment | 

0

Using snapshot functionality for mongo replication back up is not good idea while you run it. First of all you need to get the snapshot of "primary" in your replica set. but the problem is your clients can connect with a wrong write concern policy. like not asking for it or not for majority that leaves uncommitted data in your primary's oplog. even if you keep oplog or all files in local db, you may have problems when you restore (got that once, my primary was looking a rollback with no reason).

my suggestion (i assume you can not stop mongod) is using mongodump, yes it sounds stupid when you can have a snapshot. but the real point of using backups is able to restore them.

if you can't use mongodump or don't want to. Try MMS's cloud backup feature. which can restore whole replica set easily.

Edit:
Another option is running a delayed replica member. here is documentation. which covers most of the backup needs. just determine the delay time. by this way you don't eat up your server's io with running backup/archive tools.

share|improve this answer

edited Dec 28 '14 at 21:13

answered Dec 28 '14 at 0:55

risyasinrisyasin

1,378815

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  The fact is that mongo dump takes more than 24h to restore. The data size is around 300GB. Tt takes so long, that even if it needs to recreate the indexes, it does not make sense, the restore server is not running at full capacity.
  
  – Alex F
  Dec 28 '14 at 21:07
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  and about MMS, we tried, and it was great. Unfortunately, their pricing is completely inadequate for us: we do constant updates. the data size grows slowly, but the oplog operations are numerous, which is why their pricing per oplog operation is killing.
  
  – Alex F
  Dec 28 '14 at 21:12
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  well if you have hundreds of gigabytes, it changes a lot. so i have updated my answer.
  
  – risyasin
  Dec 28 '14 at 21:14
  

  
  
  
  

add a comment | 

Using snapshot functionality for mongo replication back up is not good idea while you run it. First of all you need to get the snapshot of "primary" in your replica set. but the problem is your clients can connect with a wrong write concern policy. like not asking for it or not for majority that leaves uncommitted data in your primary's oplog. even if you keep oplog or all files in local db, you may have problems when you restore (got that once, my primary was looking a rollback with no reason).

my suggestion (i assume you can not stop mongod) is using mongodump, yes it sounds stupid when you can have a snapshot. but the real point of using backups is able to restore them.

if you can't use mongodump or don't want to. Try MMS's cloud backup feature. which can restore whole replica set easily.

Edit:
Another option is running a delayed replica member. here is documentation. which covers most of the backup needs. just determine the delay time. by this way you don't eat up your server's io with running backup/archive tools.

share|improve this answer

edited Dec 28 '14 at 21:13

answered Dec 28 '14 at 0:55

risyasinrisyasin

1,378815

share|improve this answer

edited Dec 28 '14 at 21:13

answered Dec 28 '14 at 0:55

risyasinrisyasin

1,378815

answered Dec 28 '14 at 0:55

risyasinrisyasin

1,378815

answered Dec 28 '14 at 0:55

risyasinrisyasin

1,378815