One of the great perks of living in the San Francisco Bay Area is proximity to some amazing wine regions. Over the last couple years, I've visited vineyards in regions like Napa Valley, Sonoma Valley, Paso Robles, and even Malibu. I recently ran into a machine learning data set that has data on 6000 Portuguese wines that includes a 1-10 quality rating, which seems like a great excuse to build a neural network that can predict the 1-10 quality rating based on factors like residual sugar and alcohol content. Effectively, this neural network attempts to match the wine palate of whoever put this data set together.
Promise.prototype.finally() recently reached stage 4 of the TC39 proposal process. This means the
Promise.prototype.finally() proposal was accepted and is now part of the latest draft of the ECMAScript spec, and it is only a matter of time before it lands in Node.js. This article will show you how to use
Promise.prototype.finally() and how to write your own simplified polyfill.
MongoDB 3.2 introduced the
$lookup aggregation framework pipeline stage, which let you pull documents from a separate collection into your aggregation framework pipeline. Before MongoDB 3.6,
$lookup could only do left outer joins with equality matching. In other words, suppose you had a collection of users, a collection of stocks, and a collection that mapped users to the stocks they hold. The
$lookup stage can give you an array of stocks a user holds. But in MongoDB 3.2 and 3.4,
$lookup could not give you just the stocks that had gone up in price since the customer bought them.
Before MongoDB 3.6, tailing the MongoDB oplog was the only way to listen for changes to your MongoDB database. The oplog is a special collection that logs all inserts and updates that come in to your MongoDB server so other members of the replica set can copy them. Tools like Meteor and MoSQL were built on tailing the oplog. Unfortunately, the oplog was built primarily to support replication internally as opposed to being a user-facing feature. Change streams are a usability layer on top of the MongoDB oplog that make change detection in MongoDB much easier than tailing the oplog directly.
Strong support for arrays has always been one of MongoDB's killer features. For example, suppose you have a collection of blog posts where each document contains an array of comments as shown below. Before MongoDB 3.6, you could only update at most one element of the
comments array at a time because of limitations with the positional operator
$. Array filters in MongoDB 3.6 remove that limitation and add several more exciting features, like updating nested arrays.
One of the major reasons for the mongoose 5 release was the MongoDB driver removing support for the
authenticate() function. Up until Mongoose 4.11, that function was the only way mongoose supported doing authentication. In Mongoose 4.11.0, we added the annoying but necessary
useMongoClient deprecation warning, and in mongoose 5.0.0-rc0 we deleted 686 lines of legacy connection logic that removed the need for the
useMongoClient option. In addition, we made a couple related improvements to the connection API that will make mongoose much cleaner to work with: we made
mongoose.connect() always return a promise, and we added a global and connection-level
Mongoose 5.0.0-rc0 introduced several important changes to the way middleware works. The most pronounced difference is the ability to use promises and async/await with mongoose middleware. In addition, there are a couple more subtle changes that make the middleware API more consistent. In this article, I'll cover two changes. The first is that post hooks now always get flow control, even synchronous post hooks. The second is that query middleware is applied when the model is compiled for performance reasons.
This is the beginning of my first subscriber-only article on The Code Barbarian's Patreon. The article goes into much more depth than normal tech articles and has enough content to fill an ebook, so the whole article is only available to Patreon subscribers. If you like this article, please consider subscribing on Patreon to support future in-depth guides. The next article, "Building Distributed Locking with Node.js and MongoDB", ships on January 15, 2018!
Mongoose 5.0.0-rc0 was released yesterday. This is the first backwards-breaking release for mongoose since 4.0.0 was released in March 2015. The major forcing functions for this release were that MongoDB 3.6 removed support for the way mongoose did authentication (which is why mongoose needed the
useMongoClient option) and the 3.0.0 release of the official MongoDB Node.js driver. Another major priority for Mongoose 5 was dropping support for Node.js < 4 and MongoDB < 3.0.0. Both Node.js 0.12 and MongoDB 2.6 are about a year past their official end-of-life, and removing support for them has enabled us to make our code base much leaner.
Monogram has a powerful middleware system that makes cross-cutting concerns easy. One use case is denormalization, the practice of storing all or part of one document in another document. Denormalization is a powerful paradigm for both performance and data integrity. When done well, denormalization means you only need one query to fetch all necessary data, rather than using joins or
$lookup stages that may require their own indexes.
Mongoose 4.13 was released a couple weeks ago, with support for a powerful new feature: middleware for aggregation. The primary motivation for this feature was to enable plugins like mongoose-explain to work with
aggregate(), as well as enabling us to refactor discriminators to be a plugin. Aggregation middleware is a natural complement to query middleware, it lets you apply a lot of the use cases for hooks like
post('updateOne') to aggregation. In this article, I'll demonstrate using aggregation middleware to enforce soft deletes, and explain how aggregation middleware works with aggregation cursors.
Mongoose 4.13 was released last week. As usual, its packed with powerful new features. The most exciting new feature is the ability to set
ref dynamically for virtual populate. The syntax for virtual populate's dynamic properties is slightly different than the one for conventional populate, but we think this new syntax is much more powerful.
One of the most compelling reasons for async/await is the fact that async functions generally feel like simplified threads, or threads with predictable interrupts. There are numerous
I previously wrote about handling the database server going down with the MongoDB Node.js driver.
In Mongoose 4.8 we added support for
Too often, MongoDB REST API developers don't think about handling database outages until they have an outage in production. Usually you can get away with this because version 2.x of the MongoDB Node.js driver does most of the work for you: it handles attempting to reconnect and can buffer operations for you so you don't get errors during a temporary outage. However, the MongoDB Node.js driver has a lot of tunable options and corresponding subtleties that you need to be aware of. In this article, I'll cover the basics of what happens when your backend MongoDB topology goes down for single server and replica sets, so you can configure the driver to do the right thing for your use case.
You might remember locking from your undergrad systems programming class. Locks are what you use when multiple threads want to access the same chunk of memory, and you don't want one thread to clobber the other's data. You don't have threads in Node.js, but that doesn't mean you can ignore concurrency, because your Express server might get conflicting requests at roughly the same time. In this article, I'll describe how you can leverage promises and async functions to enforce the constraint that only one instance of a given function runs at a time.
I often get questions about how to secure user passwords in mongoose. The answer is a one-liner, but I now think there's a better answer: why do you store the user's password in the user document anyway? This may seem blasphemous given that MongoDB's "single view of the customer" use case and that denormalization is one of MongoDB's killer features. However, I've used this paradigm successfully for several projects. In this article, I'll make the case that a separate
AuthenticationMethod collection is the way to go for storing password hashes in your database.
Express' error handling middleware is a powerful tool for consolidating your HTTP error response logic. Odds are, if you've written Express code you've written code that looks like what you see below.
In 2015, I started working on a new ODM for MongoDB and Node.js. It was based on the now-obsolete
Object.observe() function, and I unfortunately had to scrap the whole project when the
Object.observe() proposal was unexpectedly withdrawn. A lot of the core logic from the original ODM lives on in Archetype. But my brief time in Redux land and studying falcor taught me a crucial lesson: the Model-View-Controller paradigm is not the only way to do things. MVC and ODMs still make sense, but we can build a stronger, more functional abstraction as the basis for a more concise architecture. The new monogram has now been powering 100% of Booster's API queries for weeks and it's officially production ready. Here's what monogram is all about.
Mongoose 4.11 introduced an important new option to work around a major deprecation. The
useMongoClient option is the source of the '
open() is deprecated in mongoose' deprecation warning that has caused so much discussion. This option opts you in to using Mongoose 4.11's simplified initial connection logic and allows you to avoid getting a deprecation warning from the underyling MongoDB driver.
Mongoose 4.11 was released last week and includes several neat new features. It also has one very important deprecation, so before you upgrade please read about the
useMongoClient option in the docs. I'll write more about
useMongoClient and why it is necessary another time. But first, mongoose 4.11 enables a new plugin, mongoose-lean-virtuals, which lets you apply virtuals to query results even if you use the
lean() function. This plugin is a more general version of mongoose-lean-id.
You might remember locking from your undergrad systems programming class. Locks help when multiple threads that can be interrupted at any time access a shared resource. In distributed programming (like building a Node.js server that talks to a database) you have a similar problem: parallel operations can mutate documents in the database in conflicting ways. In particular, enforcing uniqueness, such as making sure only one user has a given email, is tricky.
Mongoose setters have always had the limitation that they only work for
save(), not for queries. For example, let's say you have a schema that enforces your emails are always lowercase:
One great perk of async/await in Node.js is how well it integrates with existing libraries. By now, most popular Node.js libraries support some sort of promise-based API, so they integrate nicely with async/await. You might even have the pleasure of removing a few dependencies from your
package.json if you start using async/await instead of co. In this article, I'll show you how async/await works with mocha tests, express routes and middleware, and mongoose queries and cursors.
Mongoose 4.10.0 just landed and brings with it several powerful features and bug fixes. The most +1-ed feature in this supporting
unique in array definitions via the mongoose-unique-array plugin. This feature is implemented as a separate plugin because
mongoose-unique-array does much more than simply create a unique index, it also ties in to validators and versioning. In this article, I'll explain how to use
mongoose-unique-array and the caveats you need to be aware of when using it.
pkg is Zeit's (the company behind Next.js) new tool for bundling Node.js projects into standalone binary executables. A standalone executable has numerous advantages: as long as you're on a compatible OS, you can run the executable without installing Node.js, docker, or any other runtime. You can ship your executable to a vanilla EC2 instance and run it without any extra setup, no need to maintain AMIs or use Packer. You can also cross-compile with pkg, so you can build an OSX-compatible executable on your Linux box and vice-versa. In other words, pkg gives you the best parts of Golang in Node.js.
Next.js is a powerful framework for building server-side rendered applications. Next.js is just a wrapper around React, but it abstracts away all the ugly bits of React: build systems, transpilation, routing, CSS, etc. Most importantly, it makes server-side rendering with React dead simple, no need to carefully structure your code for use with preact-render-to-string or figure out the
lastChild pattern for replacing the server-side rendered component with the client-side rendered component. In this article, I'll show you how to build a basic Next.js app with server-side rendering on top of an Express and MongoDB API.
Async/await makes it easy to integrate asynchronous behavior with imperative constructs like for loops, if statements, and try/catch blocks. Unfortunately, it doesn't do the same for functional constructs like
Firebase is a great tool for getting data from a server to a client. Firebase handles caching, retries, socket management, and all the other unpleasant details of getting data to a client with spotty internet connection. In particular, Firebase is excellent for mobile web and mobile apps. In this article, I'll walk you through using Firebase with Preact, a lightweight React alternative, to build a simple app with server-side rendering.
Node.js 6.6.0 added a sporadically useful bug/feature: logging unhandled promise rejections to the console by default. In other words, the below script will print an error to the console:
By virtue of the event loop, scheduling tasks in Node.js is relatively straightforward. Plain old
setInterval() are sufficient for many basic use cases where you would normally use cron. However, things get more interesting when you need durable transactional scheduling, for use cases like:
I find the existence of tools like nvm baffling. I could understand if setting up Node.js required an actual installer or compiler or python, but node has pre-built binaries for most operating systems. Node and npm are both portable standalone executables, so all you need is the right binary for your OS in the right place on your file system. Especially if you're switching back and forth between Node 6.x and 7.6.0 for async/await, you should simplify your node version management workflow rather than using yet another bloated tool.
Async/await in Node.js opens up a host of powerful design patterns. Tasks that used to take complex libraries or intricate promise chaining can now be done with rudimentary
if statements and
for loops. I already wrote about these kind of design patterns with co, but async/await makes these patterns accessible in vanilla Node.js, no outside libraries required.
Arguably the biggest new feature in Node.js 7.6.0 is that the much awaited async function keyword is now available without a flag. Callback hell and promise hell are now in the past. But, like Uncle Ben always reminded us, with great power comes great responsibility, and async/await gives you a lot of new and exciting ways to shoot yourself in the foot. You still need to handle errors and be aware of the async nature of your code, otherwise you'll inevitably be complaining about "async/await hell" in 6 months.
The most important feature of most server-side frameworks is middleware: the ability to hook your own logic into the framework's logic, like IFTTT for your code. Express, Mongoose, Rails, and Django all rely heavily on middleware. LoopBack and Sequelize have hooks, AngularJS has parsers and interceptors, Hapi has extensions, but these are all just roses by another name. Except Hapi, Hapi by any other name would still have a foul code stench.
saveErrorIfNotFound option and
$where property in mongoose 4.8 gives plugins a powerful new way to modify the behavior of
save(). This feature may not seem as exciting as
eachAsync() or the major perf improvements in 4.8, but I think it will help the community develop some handy abstractions on top of
save(). How do these
MongoDB 3.2 supports 3 numeric types: 32 bit integers, 64 bit integers, and 64 bit binary floating points. MongoDB 3.4 introduces a
On a sunny Wednesday in September 2006, I showed up for my first day of an
Proxies are a powerful ES2015 feature that let you intercept operations on object properties by defining "traps" (function handlers) for getting/setting a property. For example:
Ramda is quickly becoming an indispensible part of my node projects. Lodash is more accessible and beginner-friendly, but ramda is far more powerful and expressive once you wrap your mind around it. In this article, I'll take a look at the
applySpec() function and how it can replace dependency injectors like wagner.
One particularly neat feature of mongoose is the chainable query builder API. This API provides the ability to build up MongoDB queries with helper methods, rather than via a JSON object.
Mongoose 4.5.0 introduces the ability to handle errors in middleware. This lets
If you haven't heard the React Native hype train yet, you will: it's probably
Co is a powerful tool for writing callback-free async code using vanilla ES2015. However, the sheer extent
The practice of asking brainteasers in interviews is a common punching bag
There's a lot of solutions out there for putting private projects in your
$httpBackend service is definitely one of my top 10 AngularJS features. It makes TDD incredibly easy, especially if you're testing directives as a whole.
React Native is hot right now,
Getting data into and out of MongoDB is a pain. Although MongoDB 3.0 and 3.2
I must confess, I really don't like most tech books. Except for
2015 was a big year for mongoose. Mongoose 4.0 was released, the first major
In last week's article, you
MongoDB 3.2.0 was released earlier this week. It includes some exciting
GridFS is a mechanism for
In the first two articles in this series, you built a
Strict unit tests are an interesting subject. They're simple, elegant, and do
I've spent a fair amount of time lauding Travis as an excellent go-to CI tool. But, when I wanted to use Travis
One of my favorite underused mongoose features is
Every so often I run into an npm module that uses
People have often asked me about my biggest pain points with AngularJS.
Last December, I switched my blog over to a static page generated by a vanilla NodeJS script. So far, its been a dream come true. Thanks to Cesar Devers' design skills, the blog is far more readable than the old standard WordPress template. Also, I can write my posts in Markdown, which has done wonders for my workflow. I also have more direct control over my layout and analytics, so I can use KeenIO to track some basic metrics. Getting to this new blog infrastructure was an adventure. I had brief flings with many static site generators before settling on my current setup. In this article, I'll explain why I decided to skip out on all the established tools and just write my own solution.
Recently, I've been looking into StrongLoop's LoopBack framework. LoopBack generates Express REST APIs by asking you a few simple questions at the command line. LoopBack lets you swap out different storage layers. For each model you define, you can choose to store it in MongoDB, Oracle, MySQL, or Microsoft SQL Server (or even in memory). Say you decide to store your users in MongoDB but your user's gift cards in MySQL (for transactions). Even if you started writing your code with gift cards stored in MongoDB, LoopBack's database abstraction layer makes switching a one-liner. Furthermore, LoopBack has SDKs for generating REST API clients in AngularJS, Android, and iOS. In short, LoopBack is a powerful tool for generating REST APIs that you can extend to scaffold client-side code.
Mongoose 3.9.7 has just been released. While I did say that 3.9.6 would be the
In case you haven't come across Petko Petkov's post on injection attacks against MongoDB and NodeJS yet, its definitely worth a careful read. In this article, he explains a pretty simple exploit that I suspect affects a fair number of applications, including some that I've implemented.
Two weeks ago marked a big milestone: mongoose 3.9.0 was released. Be warned, mongoose's versioning practice is that even numbered branches are stable and odd are unstable. While all our tests check out on 3.9.0, I would recommend sticking to 3.8.x releases in production for now. 3.9.0 was mongoose's first unstable release since October 2013. While the changes in 3.9.0 were relatively minor, they open the door to getting some interesting features into 4.0. Here are some of the high-level features I think should make it in to 4.0:
I have an important announcement to make: over the last couple weeks I've been taking over maintaining mongoose, the popular MongoDB/NodeJS ODM. I have some very big shoes to fill, Aaron Heckmann has done an extraordinary job building mongoose into an indispensable part of the NodeJS ecosystem. As an avid user of mongoose over the last two years, I look forward to continuing mongoose's storied tradition of making dealing with data elegant and fun. However, mongoose isn't perfect, and I'm already looking forward to the next major stable release, 4.0.0. Suggestions are most welcome, but please be patient, I'm still trying to catch up on the backlog of issues and pull requests.
From a performance perspective as well as a developer productivity perspective, MongoDB really shines when you only need to load one document to display a particular page. A traditional hard drive only needs one sequential read to load a single MongoDB document, which limits your performance overhead. In addition, much like how Nas says life is simple because all he needs is one mic, grouping all the data for a single page into one document makes understanding and debugging the page much simpler.
MongoDB shipped the newest stable version of its server, 2.6.0, this week. This new release is massive: there were about 4000 commits between 2.4 and 2.6. Unsurprisingly, the release notes are a pretty dense read and don't quite convey how cool some of these new features are. To remedy that, I'll dedicate a couple posts to putting on my NodeJS web developer hat and exploring interesting use cases for new features in 2.6. The first feature I'll dig in to is text search, or, in layman's terms, Google for your MongoDB documents.
Displaying prices in different currencies is a common internationalization task for web developers. However, this task can be a bit tricky:
In last week's blog post, I showed you how to install all of the basic tools that you need to get up and running with the MEAN Stack. Didn't catch that one and need help getting started with the MEAN Stack? You can find everything you need in Introduction to the MEAN Stack, Part One.
I've received several emails asking for instructions on how to set up a basic MEAN stack app. I'm going to take it one step further and give you guys a two-part post that will walk you through creating your first MEAN stack app- from installing the tools to actually writing the code. In Part One we'll go through the setup and installation process. Next in Part Two we'll walk through the steps for building a very simple to-do list. Part One consists of seven steps, although only the first two are are strictly necessary.
If you're familiar with Ruby on Rails and are using MongoDB to build a NodeJS app, you might miss some slick ActiveRecord features, such as declarative validation. Diving into most of the basic tutorials out there, you'll find that many basic web development tasks are more work than you like. For example, if we borrow the style of http://howtonode.org/express-mongodb, a route that pulls a document by its ID will look something like this:
This post was featured as a guest blog post for MongoDB on April 30th 2013, which can be found here