Unit 3, Lesson 1

$\textbf{MongoDB and Node}$
$\text{Written by:} $
$\text{Jacob Levine, Class of 2020}$
$\text{Last Revised: March 2020}$

MongoDB is NoSQL database software that uses a document structure similar to JSON, so it's often used on Node.js websites. Due to its powerful searching and high scalibility, Mongo is used on some of the largest applications, such as Google and Uber. It's somewhat controversial for some decisions that prioritize speed over stability.

Initialization and the Document Model

In Mongo, database entries are called "documents" and work very similarly to JS objects. These documents are stored in "collections" which are grouped into "databases". All documents have a unique "_id" property that is assigned either before (i.e. during object creation) or on insertion. For this project, we're going to use the MongoClient module. The async function below connects to the database in the connection string (in this case the 2022 scouting DB, also note that connection strings will often be stored in enviroment variables instead of source code to avoid revealing them) and pulls out a db (we'll be using the intro-to-node db later to demonstrate reading and writing).

In [1]:
const MongoClient = require('mongodb').MongoClient;
let db;
let getDB = async (dbname) => {
    try {
        const client = await MongoClient.connect("mongodb+srv://intro-to-node:epycepoch2019@2022-scouting-4vfuu.mongodb.net/test");
        db = client.db(dbname);
    } catch (err) {
        console.error(err);
    }
    return db;
};
In [2]:
getCollection = async (db,coll) => {
    let loc = await getDB(db)
    return loc.collection(coll)
};
Out[2]:
[AsyncFunction: getCollection]

find and findOne

findOne returns the first object that the database finds that matches the selector (a collection of properties that we're looking for). find goes through and returns a cursor of all of the elements that match the selector. This cursor can be for-looped through (or .forEached) or be casted to an array, like in the function below

In [3]:
findOne = async(db,coll,selector) => {
    let loc = await getCollection(db,coll)
    return loc.findOne(selector)
};
Out[3]:
[AsyncFunction: findOne]
In [4]:
findOne("intro-to-node", "col1", {"match":10, "team_scouted" : 111})
Out[4]:
{
  _id: '2020ilch10111',
  competition: '2020ilch',
  data: {
    'pass-line': 'Yes',
    'balls-started': 0,
    'balls-collected': 2,
    'balls-upper': 10,
    'balls-lower': 0,
    'shooting-vulnerable': 'Yes',
    defense: 'No',
    climb: 'Yes',
    'shooting-notes': 'They normally shot from just in front of color wheel ',
    competency: 'Good',
    speed: 'Fast',
    'strategic-focus': 'Offense',
    'strategy-notes': 'Amazing offense bot'
  },
  match: 10,
  scouter: { name: 'Dev Singh', id: '114979123360880121338' },
  team_scouted: 111
}
In [5]:
find  = async(db,coll,selector) => {
    let loc = await getCollection(db,coll)
    return loc.find(selector).toArray() //find returns a cursor, but can cast to Array
};
Out[5]:
[AsyncFunction: find]
In [6]:
find("intro-to-node", "col1", {"match":10})
Out[6]:
[
  {
    _id: '2020ilch1063',
    competition: '2020ilch',
    data: {
      'balls-started': 3,
      'pass-line': 'Yes',
      'balls-upper': 0,
      'balls-collected': 0,
      'balls-lower': 6,
      'shooting-vulnerable': 'No',
      'shooting-notes': 'They are a dumper robotics just like us ',
      climb: 'Failed',
      defense: 'No',
      'spun-wheel': 'No',
      competency: 'Meh',
      speed: 'Slow',
      'strategic-focus': 'Offense',
      'strategy-notes': 'They are a dumperbot \n\nBasically the same as ours '
    },
    match: 10,
    scouter: { name: 'Sarah Oquendo', id: '116902796378120085051' },
    team_scouted: 63
  },
  {
    _id: '2020ilch108096',
    competition: '2020ilch',
    data: {
      'balls-started': 3,
      'balls-collected': 0,
      'balls-upper': 0,
      'balls-lower': 3,
      'pass-line': 'Yes',
      'spun-wheel': 'No',
      'shooting-vulnerable': 'No',
      'shooting-notes': 'No shooter',
      defense: 'No',
      climb: 'No Attempt',
      competency: 'Awful',
      speed: 'Slow',
      'strategic-focus': 'Offense',
      'strategy-notes': "They just bring balls over. Can't do much with them"
    },
    match: 10,
    scouter: { name: 'Alexander Wells', id: '113332350115233830981' },
    team_scouted: 8096
  },
  {
    _id: '2020ilch10111',
    competition: '2020ilch',
    data: {
      'pass-line': 'Yes',
      'balls-started': 0,
      'balls-collected': 2,
      'balls-upper': 10,
      'balls-lower': 0,
      'shooting-vulnerable': 'Yes',
      defense: 'No',
      climb: 'Yes',
      'shooting-notes': 'They normally shot from just in front of color wheel ',
      competency: 'Good',
      speed: 'Fast',
      'strategic-focus': 'Offense',
      'strategy-notes': 'Amazing offense bot'
    },
    match: 10,
    scouter: { name: 'Dev Singh', id: '114979123360880121338' },
    team_scouted: 111
  },
  {
    _id: '2020ilch102338',
    competition: '2020ilch',
    data: {
      'balls-started': 2,
      'balls-collected': 1,
      'balls-upper': 14,
      'balls-lower': 0,
      'pass-line': 'Yes',
      'spun-wheel': 'Position',
      climb: 'Yes',
      defense: 'No',
      'shooting-notes': 'From front of control panel',
      competency: 'Good',
      speed: 'Fast',
      'strategy-notes': 'Good autonomous, good at collecting and shooting balls fast/well, spun the wheel (unsure if did the color part though)'
    },
    match: 10,
    scouter: { name: 'Cordelia Sirais', id: '104097601226614026776' },
    team_scouted: 2338
  },
  {
    _id: '2020ilch101756',
    competition: '2020ilch',
    data: {
      'balls-started': 2,
      'balls-collected': 2,
      'balls-upper': 15,
      'balls-lower': 0,
      'pass-line': 'Yes',
      'shooting-vulnerable': 'Yes',
      'spun-wheel': 'No',
      defense: 'No',
      climb: 'No Attempt',
      'shooting-notes': 'Rendezvous point and closer to goal',
      competency: 'Good',
      speed: 'Ludicrous',
      'strategic-focus': 'Offense',
      'strategy-notes': 'We can use them to get fast points in teleop and autonomous, but not endgame.'
    },
    match: 10,
    scouter: { name: 'Liam Nelson', id: '109531456606510077850' },
    team_scouted: 1756
  },
  {
    _id: '2020ilch106237',
    competition: '2020ilch',
    data: {
      'pass-line': 'Yes',
      'balls-started': 3,
      'balls-collected': 0,
      'balls-upper': 0,
      'balls-lower': 0,
      'spun-wheel': 'No',
      climb: 'No Attempt',
      defense: 'No',
      'shooting-vulnerable': 'No',
      competency: 'Awful',
      speed: 'Med.',
      'strategic-focus': 'Offense',
      'strategy-notes': 'Not really useful, robot seemed to stall out and sat in one place for over half of the match. Maybe use them for defense.'
    },
    match: 10,
    scouter: { name: 'Shawn Coutinho', id: '100874915124225735950' },
    team_scouted: 6237
  }
]

insertMany and insertOne

insertOne adds one object as a document to a collection while insertMany adds each element of an array of objects as its own document. Note that if you specify an _id property to a document, any of the insert properties will fail if your database already has a document with the same _id (we'll talk about updating documents below...)

In [7]:
insertOne = async(db,coll,info) => {
    let loc = await getCollection(db,coll)
    loc.insertOne(info, (err,res) => {
        if(err){
            console.error(err)
        }else{
            console.log("Number of documents inserted: " + res.insertedCount );
        }
    })
};
Out[7]:
[AsyncFunction: insertOne]
In [8]:
insertOne("intro-to-node","col1", {"match": -1, data: "Michael is lame"})
Number of documents inserted: 1
In [9]:
insert = async(db,coll,info) => {
    let loc = await getCollection(db,coll)
    loc.insertMany(info, (err,res) => {
        if(err){
            console.error(err)
        }else{
            console.log("Number of documents inserted: " + res.insertedCount);
        }
    });
};
Out[9]:
[AsyncFunction: insert]
In [10]:
insert("intro-to-node","col1", [{"match" : -2, "class" : "node intro"}, {"match" : -1, "MichaelIsLame" : true}])
Number of documents inserted: 2

remove

The remove function removes all documents that match a selector. There is no removeOne function, however, you can get it to remove only the first document it finds by setting the single (called justOne in MongoDB shell and some other languages) property to true. Below, I have added a callback that returns the number of documents removed:

In [11]:
remove = async(db,coll,selector) => {
    let loc = await getCollection(db,coll)
    loc.remove(selector, (err,res) => {
        if(err){
            console.error(err)
        }else{
            console.log("Number of documents removed: " + res.result.n);
        }
    });
};
Out[11]:
[AsyncFunction: remove]
In [12]:
remove("intro-to-node", "col1", {"match" : -1})
Number of documents removed: 2
In [13]:
removeOne = async(db,coll,selector) => {
    let loc = await getCollection(db,coll)
    loc.remove(selector, {"single":true}, (err,res) => {
        if(err){
            console.error(err)
        }else{
            console.log("Number of documents removed: " + res.result.n);
        }
    });
};
Out[13]:
[AsyncFunction: removeOne]
In [14]:
removeOne("intro-to-node", "col1", {"match":-2})
Number of documents removed: 1

update and upsert

updateOne replaces one value that matches the selector with a replacement object. It throws an error if no documents match the selector, however, this behavior can be changed with upsert. Setting upsert to true makes the db add a new document if there is nothing to replace.

updateOne = async(db,coll,selector,replacement) => { let loc = await getCollection(db,coll) loc.update(selector, replacement); };

In [15]:
updateOne = async(db,coll,selector,replacement) => {
    let loc = await getCollection(db,coll)
    loc.update(selector, replacement);
};
Out[15]:
[AsyncFunction: updateOne]
In [16]:
updateOne("intro-to-node", "col1", {"match" : -2}, {"match" : -2, "class" : "introToNode"})
In [17]:
updateOnewithUpsert = async(db,coll,selector,replacement) => {
    let loc = await getCollection(db,coll)
    loc.update(selector, replacement, {"upsert": true});
};
Out[17]:
[AsyncFunction: updateOnewithUpsert]
In [18]:
updateOnewithUpsert("intro-to-node", "col1", {"match" : -3}, {"match" : -3, "node-mongo":true})

Selectors

Mongo has a complex system of selectors and regex searches to find documents. We're not going to cover all of them, and I would reccomend that you check out the documentation for more, but here are some of the most common ones:

Comparison

  • $lt- less than
  • $gt- greater than
  • $lte- less than or equal to
  • $gte- greater than or equal to
  • $eq- equal to
  • $gt- not equal to

** for example, to get all data for matches after 75:

In [19]:
find("intro-to-node", "col1", {"match" : {"$gt" : 75}})
Out[19]:
[
  {
    _id: '2020ilch768014',
    competition: '2020ilch',
    data: {
      'balls-started': 1,
      'balls-collected': 0,
      'balls-upper-auto': 0,
      'balls-lower-auto': 0,
      'pass-line': 'Yes',
      'spun-wheel': 'No',
      'balls-upper-teleop': 0,
      'balls-lower-teleop': 0,
      'shooting-vulnerable': 'Yes',
      'shooting-notes': 'Rendezvous point',
      climb: 'No Attempt',
      defense: 'Yes',
      'defense-notes': '111',
      competency: 'Awful',
      speed: 'Slow',
      'strategy-notes': 'They went into the opponents’ loading bay and their drivetrain likely broke.'
    },
    match: 76,
    scouter: { name: 'Liam Nelson', id: '109531456606510077850' },
    team_scouted: 8014
  },
  {
    _id: '2020ilch772136',
    competition: '2020ilch',
    data: {
      'balls-started': 2,
      'balls-collected': 0,
      'balls-upper-auto': 1,
      'balls-lower-auto': 0,
      'pass-line': 'Yes',
      'spun-wheel': 'No',
      'balls-upper-teleop': 0,
      'balls-lower-teleop': 0,
      'shooting-vulnerable': 'No',
      'shooting-notes': 'No shooting',
      climb: 'No Attempt',
      defense: 'No',
      competency: 'Awful',
      speed: 'Slow',
      'strategy-notes': 'They can be a defense bot, but they probably aren’t fast enough to do that well.'
    },
    match: 77,
    scouter: { name: 'Liam Nelson', id: '109531456606510077850' },
    team_scouted: 2136
  },
  {
    _id: '2020ilch782151',
    competition: '2020ilch',
    data: {
      'balls-started': 3,
      'balls-collected': 0,
      'balls-upper-auto': 0,
      'balls-lower-auto': 0,
      'pass-line': 'Yes',
      'balls-upper-teleop': 0,
      'balls-lower-teleop': 0,
      'spun-wheel': 'No',
      'shooting-vulnerable': 'No',
      climb: 'No Attempt',
      defense: 'No',
      'shooting-notes': 'No successful shooting done',
      competency: 'Awful',
      speed: 'Slow',
      'strategic-focus': 'Offense',
      'strategy-notes': 'They can act as a defense or feeder bot for us.'
    },
    match: 78,
    scouter: { name: 'Liam Nelson', id: '109531456606510077850' },
    team_scouted: 2151
  },
  {
    _id: '2020ilch887608',
    competition: '2020ilch',
    data: {
      'pass-line': 'Yes',
      'balls-started': 1,
      'balls-collected': 2,
      'balls-upper-auto': 2,
      'balls-lower-auto': 5
    },
    match: 88,
    scouter: { name: 'Dev Singh', id: '114979123360880121338' },
    team_scouted: 7608
  }
]

Array Selectors

  • $elemMatch- at least one element matches the query
  • $size- array is a certain size
In [20]:
updateOnewithUpsert("intro-to-node","col1", {"names" : ["Jacob Levine", "Dev Singh", "Arthur Lu", "Ian Fowler"]},{"names" : ["Jacob Levine", "Dev Singh", "Arthur Lu", "Ian Fowler"]})
In [21]:
findOne("intro-to-node","col1", {"names" : {"$elemMatch":{"$eq":"Jacob Levine"}}})
Out[21]:
{
  _id: 5e8365ee3dac8326c4bcb60b,
  names: [ 'Jacob Levine', 'Dev Singh', 'Arthur Lu', 'Ian Fowler' ]
}

Logical Operators

  • $and- only if all querys are true
  • $or- at least one query is true
  • $not- only if query isn't true
  • $nor- only if all queries are not true
In [22]:
find("intro-to-node", "col1", {"$and":[{"team_scouted": 2022}, {"match" : {"$lte" : 20}}]})
Out[22]:
[
  {
    _id: '2020ilch72022',
    competition: '2020ilch',
    data: {
      'balls-started': 2,
      'pass-line': 'Yes',
      'balls-collected': 0,
      'balls-upper': 0,
      'balls-lower': 2,
      defense: 'No',
      'defense-notes': '4096',
      climb: 'No Attempt',
      competency: 'Meh',
      speed: 'Fast',
      'strategic-focus': 'Defense',
      'strategy-notes': 'Defense, early low, climb'
    },
    match: 7,
    scouter: { name: 'Jacob Levine', id: '118006453012298349911' },
    team_scouted: 2022
  },
  {
    _id: '2020ilch122022',
    competition: '2020ilch',
    data: {
      'pass-line': 'Yes',
      'balls-lower-teleop': 12,
      climb: 'Yes',
      'balls-lower-auto': 3,
      defense: 'No',
      'spun-wheel': 'No',
      'shooting-vulnerable': 'No',
      'balls-upper-teleop': 0,
      'balls-upper-auto': 0,
      'balls-started': 0,
      'balls-collected': 0,
      competency: 'Good',
      speed: 'Fast',
      'strategic-focus': 'Offense',
      'strategy-notes': 'We are this robot'
    },
    match: 12,
    scouter: { name: 'Jacob Levine', id: '118006453012298349911' },
    team_scouted: 2022
  },
  {
    _id: '2020ilch192022',
    competition: '2020ilch',
    data: {
      'balls-started': 3,
      'balls-upper': 0,
      'balls-lower': 13,
      'balls-collected': 0,
      'pass-line': 'Yes',
      'spun-wheel': 'No',
      'shooting-vulnerable': 'No',
      climb: 'Failed',
      defense: 'No',
      competency: 'Best',
      speed: 'Fast',
      'strategic-focus': 'Offense'
    },
    match: 19,
    scouter: { name: 'Shawn Coutinho', id: '100874915124225735950' },
    team_scouted: 2022
  }
]
In [ ]: