With MongoDB, you don't usually need to create collections explicitly - they just come to exist the moment you try to store an object in one.

This suggestion is likely overkill, but I thought I might as well suggest it anyway just for thought: you might consider using mongoengine, which is an "OM" (like an ORM, but Mongo isn't relational… ☺).

On the other hand, it looks like you are doing something extremely simple here so probably not worth it.

Looks like a reasonable replacement for Redis to me, but doesn't Redis also have the ability to persist its data to disk? I.e., couldn't you accomplish the goal by just configuring Redis to persist?

Thanks for reviewing, sorry for not seeing it earlier!

About the collection creation, I explicitly create it because I want to set it as "capped", i.e. as a bounded queue. This way Mongo will deal with the queue limits by itself.

Yeah, I thought about using a model library but I'm just inserting and pulling all items, so I'll just use the basic driver. Thanks for the pointer though :-)

About Redis, you can configure it to persist on disk, but the data will still be loaded entirely in RAM, and that's what I want to avoid here (using up all the RAM).

Hello @abompard!

In my experience, MongoDB does use all the RAM on your box too. It uses memory-mapped files, which tends to just eat up RAM until there is no more. It's usually best to give it a system-limited memory cap to prevent this, either by running it in a cgroup or by giving it its own VM. Either way, you don't want to run it on the same box as your application server without taking some action to ensure it doesn't consume all the resources.

Ah alright, thanks for pointing it out. I've read up a bit on the subject and ended up with this conclusion:

• Redis can't store more data than what's available in the RAM. The only way to limit it is to auto-evict data, and we don't want that.
• MongoDB couldn't be limited in the amount of RAM it uses (unless with cgroups), but it can store more data than the RAM. However, since version 3.2, it seems to be possible to limit the RAM and there's even a reasonable default. We will probably want to set the limit lower than 60% of the RAM, but it's not horrible.

So apparently moving to MongoDB for feed data would be useful. What do you think?

@abompard Whoah I didn't know about that new MongoDB setting. I would recommend some testing to ensure that it works as expected, but I do think you are right that it would be useful either way. If we need to, we can fall back to cgroups. You are right about how MongoDB can at least handle more data than the RAM, so I think that's enough of a reason right there.

Cool! Do you think the PR is ready to merge?

@abompard Go for it!