When running MongoDB in production, you've may see queries that should be fast, but instead are exceedingly slow. For example, my Node.js apps have seen a
findOne() on a collection with only 1 document take over 1 second.
Mongoose 5.5 was released earlier this week. This release includes 12 new features and a performance improvement. The two features I'm most excited about are hooks for user-defined static functions and the ability to pass a function to populate's
match option. In this article, I'll introduce these two new features and show how they can save you some design headache.
There's some confusion on the internet about what happens when you call
Model.find() in Mongoose. Make no mistake,
Model.find() does what you expect: find all documents that match a query. But there's some confusion about
Query#find(), setting options, promise support. In this article, I'll provide a conceptual overview of what happens when you call
Model.find() so you can answer similar questions for yourself.
Mongoose 5.4 was released on December 14, with 13 new features. The overarching theme for 5.4 is making Mongoose SchemaTypes more configurable, but that isn't the whole story. Mongoose 5.4 has several new features that will help you make your apps more robust and concise. In this article, I'll cover the new
Model.events property and the new
count option for virtual populate.
In August, the Semmle Security Research Team found a security vulnerability affecting all versions of Mongoose before 5.2.12 and 4.13.17. We released a fix on August 30 and encouraged everyone to upgrade via Twitter, our Slack channel, and our Gitter chat. HackerOne recently released a formal disclosure of this issue on November 30. This blog post constitutes Mongoose's full disclosure after a period of responsible disclosure. In this article, I'll describe the vulnerability and how Mongoose patched it.
Mongoose 5.3.0 was shipped on September 28. This minor release includes 17 new features and improvements that will help you clean up repetitive code. In particular, the
orFail() query helper and the new
toJSON global options will make life much easier for devs building Express APIs and backend services with Mongoose. In this article, I'll provide an overview of these two features and what makes them so useful.
One of the major challenges when working with AWS Lambda is bundling all your
node_modules into one zip file. Most simple examples rely on zipping up the entirety of
./node_modules, but that doesn't scale well if you're looking to built a suite of Lambda functions as opposed to a single "Hello, World" example. In this article, I'll demonstrate the problem with zipping up Lambda functions yourself and show you how to use Webpack to bundle a Lambda function that connects to MongoDB.
Let's be frank: backwards-breaking API changes are painful. Especially when
MongoDB Stitch is MongoDB's backend-as-a-service solution. Stitch supports a wide variety of high-level features, like Twilio integration and field-level access control, but it also supports custom functions similar to Amazon Web Services' Lambda. In this article, I'll walk you through setting up backend application in MongoDB Stitch on top of MongoDB Atlas. This backend application regularly ingests stock price data from the IEX API and exposes an REST API endpoint for loading a stock's latest price.
One of the most exciting features of MongoDB Stitch is the ability to read and write
Transactions are undoubtedly the most
Transactions are undoubtedly the most exciting new feature in MongoDB 4.0.
Mongoose 5.1.0 was released on May 10, and introduced 10 new features. The feature I'm most
Serverless architectures are becoming increasingly popular, and with good
Serverless architectures are becoming increasingly popular, and with good
Serverless architectures are becoming increasingly popular, but, when using
MongoDB 3.2 introduced the
$lookup aggregation framework pipeline stage, which let you pull documents from a separate collection into your aggregation framework pipeline. Before MongoDB 3.6,
$lookup could only do left outer joins with equality matching. In other words, suppose you had a collection of users, a collection of stocks, and a collection that mapped users to the stocks they hold. The
$lookup stage can give you an array of stocks a user holds. But in MongoDB 3.2 and 3.4,
$lookup could not give you just the stocks that had gone up in price since the customer bought them.
Before MongoDB 3.6, tailing the MongoDB oplog was the only way to listen for changes to your MongoDB database. The oplog is a special collection that logs all inserts and updates that come in to your MongoDB server so other members of the replica set can copy them. Tools like Meteor and MoSQL were built on tailing the oplog. Unfortunately, the oplog was built primarily to support replication internally as opposed to being a user-facing feature. Change streams are a usability layer on top of the MongoDB oplog that make change detection in MongoDB much easier than tailing the oplog directly.
Strong support for arrays has always been one of MongoDB's killer features. For example, suppose you have a collection of blog posts where each document contains an array of comments as shown below. Before MongoDB 3.6, you could only update at most one element of the
comments array at a time because of limitations with the positional operator
$. Array filters in MongoDB 3.6 remove that limitation and add several more exciting features, like updating nested arrays.
One of the major reasons for the mongoose 5 release was the MongoDB driver removing support for the
authenticate() function. Up until Mongoose 4.11, that function was the only way mongoose supported doing authentication. In Mongoose 4.11.0, we added the annoying but necessary
useMongoClient deprecation warning, and in mongoose 5.0.0-rc0 we deleted 686 lines of legacy connection logic that removed the need for the
useMongoClient option. In addition, we made a couple related improvements to the connection API that will make mongoose much cleaner to work with: we made
mongoose.connect() always return a promise, and we added a global and connection-level
Mongoose 5.0.0-rc0 introduced several important changes to the way middleware works. The most pronounced difference is the ability to use promises and async/await with mongoose middleware. In addition, there are a couple more subtle changes that make the middleware API more consistent. In this article, I'll cover two changes. The first is that post hooks now always get flow control, even synchronous post hooks. The second is that query middleware is applied when the model is compiled for performance reasons.
This is the beginning of my first subscriber-only article on The Code Barbarian's Patreon. The article goes into much more depth than normal tech articles and has enough content to fill an ebook, so the whole article is only available to Patreon subscribers. If you like this article, please consider subscribing on Patreon to support future in-depth guides. The next article, "Building Distributed Locking with Node.js and MongoDB", ships on January 15, 2018!
Mongoose 5.0.0-rc0 was released yesterday. This is the first backwards-breaking release for mongoose since 4.0.0 was released in March 2015. The major forcing functions for this release were that MongoDB 3.6 removed support for the way mongoose did authentication (which is why mongoose needed the
useMongoClient option) and the 3.0.0 release of the official MongoDB Node.js driver. Another major priority for Mongoose 5 was dropping support for Node.js < 4 and MongoDB < 3.0.0. Both Node.js 0.12 and MongoDB 2.6 are about a year past their official end-of-life, and removing support for them has enabled us to make our code base much leaner.
Monogram has a powerful middleware system that makes cross-cutting concerns easy. One use case is denormalization, the practice of storing all or part of one document in another document. Denormalization is a powerful paradigm for both performance and data integrity. When done well, denormalization means you only need one query to fetch all necessary data, rather than using joins or
$lookup stages that may require their own indexes.
Mongoose 4.13 was released a couple weeks ago, with support for a powerful new feature: middleware for aggregation. The primary motivation for this feature was to enable plugins like mongoose-explain to work with
aggregate(), as well as enabling us to refactor discriminators to be a plugin. Aggregation middleware is a natural complement to query middleware, it lets you apply a lot of the use cases for hooks like
post('updateOne') to aggregation. In this article, I'll demonstrate using aggregation middleware to enforce soft deletes, and explain how aggregation middleware works with aggregation cursors.
Mongoose 4.13 was released last week. As usual, its packed with powerful new features. The most exciting new feature is the ability to set
ref dynamically for virtual populate. The syntax for virtual populate's dynamic properties is slightly different than the one for conventional populate, but we think this new syntax is much more powerful.
I previously wrote about handling the database server going down with the MongoDB Node.js driver.
In Mongoose 4.8 we added support for
Too often, MongoDB REST API developers don't think about handling database outages until they have an outage in production. Usually you can get away with this because version 2.x of the MongoDB Node.js driver does most of the work for you: it handles attempting to reconnect and can buffer operations for you so you don't get errors during a temporary outage. However, the MongoDB Node.js driver has a lot of tunable options and corresponding subtleties that you need to be aware of. In this article, I'll cover the basics of what happens when your backend MongoDB topology goes down for single server and replica sets, so you can configure the driver to do the right thing for your use case.
You might remember locking from your undergrad systems programming class. Locks are what you use when multiple threads want to access the same chunk of memory, and you don't want one thread to clobber the other's data. You don't have threads in Node.js, but that doesn't mean you can ignore concurrency, because your Express server might get conflicting requests at roughly the same time. In this article, I'll describe how you can leverage promises and async functions to enforce the constraint that only one instance of a given function runs at a time.
I often get questions about how to secure user passwords in mongoose. The answer is a one-liner, but I now think there's a better answer: why do you store the user's password in the user document anyway? This may seem blasphemous given that MongoDB's "single view of the customer" use case and that denormalization is one of MongoDB's killer features. However, I've used this paradigm successfully for several projects. In this article, I'll make the case that a separate
AuthenticationMethod collection is the way to go for storing password hashes in your database.
This article originally appeared on LunchBadger. LunchBadger helps you build APIs in a continuous lifecycle using serverless open source.
In 2015, I started working on a new ODM for MongoDB and Node.js. It was based on the now-obsolete
Object.observe() function, and I unfortunately had to scrap the whole project when the
Object.observe() proposal was unexpectedly withdrawn. A lot of the core logic from the original ODM lives on in Archetype. But my brief time in Redux land and studying falcor taught me a crucial lesson: the Model-View-Controller paradigm is not the only way to do things. MVC and ODMs still make sense, but we can build a stronger, more functional abstraction as the basis for a more concise architecture. The new monogram has now been powering 100% of Booster's API queries for weeks and it's officially production ready. Here's what monogram is all about.
Mongoose 4.11 introduced an important new option to work around a major deprecation. The
useMongoClient option is the source of the '
open() is deprecated in mongoose' deprecation warning that has caused so much discussion. This option opts you in to using Mongoose 4.11's simplified initial connection logic and allows you to avoid getting a deprecation warning from the underyling MongoDB driver.
Mongoose 4.11 was released last week and includes several neat new features. It also has one very important deprecation, so before you upgrade please read about the
useMongoClient option in the docs. I'll write more about
useMongoClient and why it is necessary another time. But first, mongoose 4.11 enables a new plugin, mongoose-lean-virtuals, which lets you apply virtuals to query results even if you use the
lean() function. This plugin is a more general version of mongoose-lean-id.
You might remember locking from your undergrad systems programming class. Locks help when multiple threads that can be interrupted at any time access a shared resource. In distributed programming (like building a Node.js server that talks to a database) you have a similar problem: parallel operations can mutate documents in the database in conflicting ways. In particular, enforcing uniqueness, such as making sure only one user has a given email, is tricky.
Mongoose setters have always had the limitation that they only work for
save(), not for queries. For example, let's say you have a schema that enforces your emails are always lowercase:
One great perk of async/await in Node.js is how well it integrates with existing libraries. By now, most popular Node.js libraries support some sort of promise-based API, so they integrate nicely with async/await. You might even have the pleasure of removing a few dependencies from your
package.json if you start using async/await instead of co. In this article, I'll show you how async/await works with mocha tests, express routes and middleware, and mongoose queries and cursors.
Mongoose 4.10.0 just landed and brings with it several powerful features and bug fixes. The most +1-ed feature in this supporting
unique in array definitions via the mongoose-unique-array plugin. This feature is implemented as a separate plugin because
mongoose-unique-array does much more than simply create a unique index, it also ties in to validators and versioning. In this article, I'll explain how to use
mongoose-unique-array and the caveats you need to be aware of when using it.
pkg is Zeit's (the company behind Next.js) new tool for bundling Node.js projects into standalone binary executables. A standalone executable has numerous advantages: as long as you're on a compatible OS, you can run the executable without installing Node.js, docker, or any other runtime. You can ship your executable to a vanilla EC2 instance and run it without any extra setup, no need to maintain AMIs or use Packer. You can also cross-compile with pkg, so you can build an OSX-compatible executable on your Linux box and vice-versa. In other words, pkg gives you the best parts of Golang in Node.js.
Next.js is a powerful framework for building server-side rendered applications. Next.js is just a wrapper around React, but it abstracts away all the ugly bits of React: build systems, transpilation, routing, CSS, etc. Most importantly, it makes server-side rendering with React dead simple, no need to carefully structure your code for use with preact-render-to-string or figure out the
lastChild pattern for replacing the server-side rendered component with the client-side rendered component. In this article, I'll show you how to build a basic Next.js app with server-side rendering on top of an Express and MongoDB API.
By virtue of the event loop, scheduling tasks in Node.js is relatively straightforward. Plain old
setInterval() are sufficient for many basic use cases where you would normally use cron. However, things get more interesting when you need durable transactional scheduling, for use cases like:
saveErrorIfNotFound option and
$where property in mongoose 4.8 gives plugins a powerful new way to modify the behavior of
save(). This feature may not seem as exciting as
eachAsync() or the major perf improvements in 4.8, but I think it will help the community develop some handy abstractions on top of
save(). How do these
MongoDB 3.2 supports 3 numeric types: 32 bit integers, 64 bit integers, and 64 bit binary floating points. MongoDB 3.4 introduces a
MongoDB's powerful built-in geospatial queries are one of the big reasons
One particularly neat feature of mongoose is the chainable query builder API. This API provides the ability to build up MongoDB queries with helper methods, rather than via a JSON object.
Mongoose 4.5.0 introduces the ability to handle errors in middleware. This lets
Getting data into and out of MongoDB is a pain. Although MongoDB 3.0 and 3.2
2015 was a big year for mongoose. Mongoose 4.0 was released, the first major
In last week's article, you
MongoDB 3.2.0 was released earlier this week. It includes some exciting
GridFS is a mechanism for
Discriminators are a powerful yet unfortunately poorly documented
The NBA Finals just concluded last week, and LeBron James officially has
Mongoose 3.9.7 has just been released. While I did say that 3.9.6 would be the
In case you haven't come across Petko Petkov's post on injection attacks against MongoDB and NodeJS yet, its definitely worth a careful read. In this article, he explains a pretty simple exploit that I suspect affects a fair number of applications, including some that I've implemented.
Two weeks ago marked a big milestone: mongoose 3.9.0 was released. Be warned, mongoose's versioning practice is that even numbered branches are stable and odd are unstable. While all our tests check out on 3.9.0, I would recommend sticking to 3.8.x releases in production for now. 3.9.0 was mongoose's first unstable release since October 2013. While the changes in 3.9.0 were relatively minor, they open the door to getting some interesting features into 4.0. Here are some of the high-level features I think should make it in to 4.0:
I have an important announcement to make: over the last couple weeks I've been taking over maintaining mongoose, the popular MongoDB/NodeJS ODM. I have some very big shoes to fill, Aaron Heckmann has done an extraordinary job building mongoose into an indispensable part of the NodeJS ecosystem. As an avid user of mongoose over the last two years, I look forward to continuing mongoose's storied tradition of making dealing with data elegant and fun. However, mongoose isn't perfect, and I'm already looking forward to the next major stable release, 4.0.0. Suggestions are most welcome, but please be patient, I'm still trying to catch up on the backlog of issues and pull requests.
From a performance perspective as well as a developer productivity perspective, MongoDB really shines when you only need to load one document to display a particular page. A traditional hard drive only needs one sequential read to load a single MongoDB document, which limits your performance overhead. In addition, much like how Nas says life is simple because all he needs is one mic, grouping all the data for a single page into one document makes understanding and debugging the page much simpler.
MongoDB shipped the newest stable version of its server, 2.6.0, this week. This new release is massive: there were about 4000 commits between 2.4 and 2.6. Unsurprisingly, the release notes are a pretty dense read and don't quite convey how cool some of these new features are. To remedy that, I'll dedicate a couple posts to putting on my NodeJS web developer hat and exploring interesting use cases for new features in 2.6. The first feature I'll dig in to is text search, or, in layman's terms, Google for your MongoDB documents.
As much as I love geeking out about basketball stats, I want to put a MongoDB data set out there that's a bit more app-friendly: the USDA SR25 nutrient database. You can download this data set from my S3 bucket here, and plug it into your MongoDB instance using mongorestore. I'm very meticulous about nutrition and have, at times, kept a food journal, but sites like FitDay and DailyBurn have far too much spam and are far too poorly designed to be a viable option. With this data set, I plan on putting together an open source web-based food journal in the near future. However, I encourage you to use this data set to build your own apps.
When you are looking to run analytics on large and complex data sets, you might instinctively reach for Hadoop. However, if your data's in MongoDB, using the Hadoop connector seems like overkill if your data fits on your laptop. Luckily, MongoDB's built-in aggregation framework offers a quick solution for running sophisticated analytics right from your MongoDB instance without needing any extra setup.
In last week's blog post, I showed you how to install all of the basic tools that you need to get up and running with the MEAN Stack. Didn't catch that one and need help getting started with the MEAN Stack? You can find everything you need in Introduction to the MEAN Stack, Part One.
I've received several emails asking for instructions on how to set up a basic MEAN stack app. I'm going to take it one step further and give you guys a two-part post that will walk you through creating your first MEAN stack app- from installing the tools to actually writing the code. In Part One we'll go through the setup and installation process. Next in Part Two we'll walk through the steps for building a very simple to-do list. Part One consists of seven steps, although only the first two are are strictly necessary.
If you're familiar with Ruby on Rails and are using MongoDB to build a NodeJS app, you might miss some slick ActiveRecord features, such as declarative validation. Diving into most of the basic tutorials out there, you'll find that many basic web development tasks are more work than you like. For example, if we borrow the style of http://howtonode.org/express-mongodb, a route that pulls a document by its ID will look something like this:
If you've ever tried to build any kind of website, odds are you've had to create some way of validating and saving input from a form. Back in the bad old days this used to be a huge pain, because there were no good frameworks to help get the job done right. The three primary pain points that you have to deal with when trying to validate a form without the aid of a framework are:
This post was featured as a guest blog post for MongoDB on April 30th 2013, which can be found here