IOT PROJECT DEVELOPMENT: TECHNOLOGIES, PROBLEMS, OPTIMIZATION

Why Node.js is the Best IoT Solution

In everyday life, we are increasingly finding smart devices, such as smartwatches, microwaves, washing machines, and even homes. This is all in harmony with your smartphone, through which, by installing it on your mobile app, you can control the dozens of processes that your smart devices perform. This is where IoT magic begins, allowing you to connect your mobile app with all the smart home technology in real-time.

We have a number of programming languages ​​to solve such tasks, including Python, Java, C ++, and Ruby, which in general can do the job, and even quite well. But there is already a programming language that can work well with IoT, it's JavaScript. But even she is not a standard. Here, too, Node.js joins the game with its multimillion-dollar community, with a large number of libraries and the ability to work steadily on high-load projects with minimal use of server resources.

Node.js is a server-side JavaScript code runtime commonly used for large and multi-tasking projects. It is extremely powerful and at the same time very easy to learn a tool that is easy to understand for a developer far from JavaScript. First of all, we want to dispel a number of myths about Node.js created by those users who obviously haven't looked at the Changelog Git repository for a long time and have no idea how they have developed this technology.

  1. Most often, there is a claim that JavaScript is not an object-oriented language. This definition would have been correct 10 years ago. With the release of ECMAScript 6, we have received full class support and inheritance. Of course, this is not the same OOP that we are used to seeing in other programming languages, but no one has planned (yet) typing in JavaScript or access modifiers. Of course, for those who still want to have these capabilities by default, there is TypeScript.

  2. One of the myths is that JavaScript has no architecture and developers write the code as they wish. Unfortunately, this can be said not only about JavaScript, but also about some fairly powerful programming languages. One way or another, developers have long supported programming and design patterns and are happy to use them in their projects. For Node.js there are a number of quite popular frameworks with their own architecture and rules, so the developer somehow follows the principles of development based on these frameworks.

We would also like to list the reasons why Node.js is the best solution for developing IoT projects:

Most IoT projects use MQTT and standard WebSockets, which is well supported by Node.js.

Batch Manager (NPM) has a large number of libraries for the development of IoT projects and a large number of modules for working with devices, including Intel IoT Edison, Raspberry Pi or Arduino.

Because IoT devices are sensors, sensors, transmitters, etc., they usually generate a large amount of data and thus a large number of requests. Node.js can handle this easily with read and write thread support.

 

The main issues facing an IoT developer

The basic concept of the Internet of Things is the connection between billions of autonomous devices. Statista 2017 data shows that there are more than 20 billion devices, and by 2025, more than 70 billion of them are all connected to the network, and they are transmitting or receiving the data necessary for their proper functioning. This data is prey for other unscrupulous developers trying to somehow intercept the data stream and use it for their own purposes.

For IoT devices, security lies in the correct and thoughtful architecture of the code, the right of access to various functionalities, and the authentication of the devices or users. One of the solutions that helps prevent such problems is competent certification. Services such as AWS or the like have the most flexible way to set up certificates and permissions. Usually, the developer neglects it to find out how it works and uses one certificate for everything. In addition, this certificate has unlimited access to resources and the ability to intercept any data on the system.

Certification should be the first point at the beginning of the development of IoT projects. This is a simple task that does not take a long time. But if you think about this step incorrectly, there will be great security issues that will come to light after a cyberattack has passed and most devices have failed.

The big problem, strange as it may sound, is AI. Because of the need to process large amounts of data and perform automated actions on their basis, developers are leaning towards AI and writing algorithms that can perform system-specific tasks for the IoT platform. In this case, we are faced with a situation where we need to constantly modify these algorithms, and somehow the developer cannot anticipate all the consequences of the code that he just downloaded into the system. With a small human intervention, some variables may inadvertently have a strong correlation with little actual predictive effect.

Equally important is the misunderstanding of IoT principles. The developer sometimes studies only the basic properties of IoT and already begins to develop a fairly large IoT project, which in the end can both significantly harm the company and undermine the authority of the developer.

 

Where to start developing your IoT project on Node.js

The first question that always arises is: where to start? Design the architecture or select all the modules you need? Or start from a database? In fact, everything is much simpler than it seems. First of all, you need to draw a general database architecture and all fields. This is required to optimize the project without which the developer runs the risk of getting slow requests or even staying in the middle of development due to poor system architecture.

The database in IoT projects is the most vulnerable place, which will then be difficult to fix. The architecture should be as flexible as possible and no less optimized for the integration of new functionality. The best solution for this task is to use the HMVC (Hierarchical Model-View-Controller) pattern, where each component of the system performs its role and has a familiar MVC architecture.


After you have come up with database architecture, you need to think about how the data will be updated, at what frequency, and at what number. Usually IoT services answer this question, for example, AWS IoT has a feature, such as Rules, by which a developer or DevOps can easily configure where to store data, which database and which script to run to further process these data. It is very important to understand the full functionality of the service that you are using so that you do not continue to develop what has long been written and provided to simplify a project that is too complex.

 

How to choose the right database for IoT

Modern databases can support IoT projects in one way or another, so it all depends on how the developer implements that connection. Before choosing a database, you need to understand whether this system is suitable for the functionality that will be passed to users. Below I have tried to list the main functionality that an IoT database should have:

  • flexible architecture and index support;
  • withstanding high load;
  • user-friendly scheme;
  • support for export and import of data;
  • multifunctional search on various parameters;
  • security;
  • value for money.

 

When the IoT project begins, everyone chooses for themselves which items will be most important for the development of high-load projects. No one prohibits the use of multiple databases, and in fact, it is not a bad practice. Projects can contain such complex architecture that one base is not able to provide all the necessary functionality. However, it should be understood that you do not need to use a powerful enough database for one simple task, but you should find a more specialized database and use it.

InfluxDB was launched in 2013 and remains one of the best databases for IoT today. This is a time-series open source database written in the Go programming language. This solution is perfectly optimized for IoT, which is what makes it so popular today

CrateDB is a SQL database distribution. It is open-source and runs on Java. Its main feature is that it already has a set of libraries: Facebook Presto, Apache Lucene or Elasticsearch, which make it easier to work with the database and do not need additional settings. This database is specially designed to be integrated into IoT platforms of varying complexity.

MongoDB is an open-source NoSQL database. For every developer who has written at least one project on Node.js, MongoDB has become the best companion for solving hundreds of different data problems. It is easy to learn and has a JSON-like architecture that combines perfectly with JavaScript. This database is not as suitable for IoT as it is for integration into the IoT platform, which is written in Node.js.

Amazon DynamoDB is a time-series, closed-source database that belongs to Amazon Web Services. Like other analogs, such as Cloud Bigtable or IBM Informix, it should be used in conjunction with other services offered by the company. In this case, AWS IoT, together with DynamoDB, creates the most efficient platform for the development of high-load IoT projects. Separate use of this database is also a good solution, but there are many cheaper analogs on the market that will produce the same result.

What can we conclude by looking at various examples of modern IoT databases? All of them are great for developing IoT platforms, but each of the databases on offer has its own widgets that allow the developer to choose the one that suits him or her best for the needs of the IoT project.

 

Proper optimization of Big Data in IoT projects

We have selected the main technology stack in our IoT project and have considered the database server architecture. It's time to start working out. First, determine which part of the project will be the maximum load and how to optimize this part. If we use a comprehensive solution, such as AWS IoT and DynamoDB, in this case, the load takes over AWS, which makes it possible to not think about the inability to process large amounts of data. On the other hand, if we build the architecture on our own server, here are some solutions to help us with this.

First of all, there is an opportunity to expand our future architecture as the project grows, and usually, the developer tends to use Docker containers and Load Balancer, which will automatically help scale the project. No matter what you choose - whether it's Amazon ECS or HAProxy - you need to know how this technology works, how to set it up and how to maintain it. The main problems faced by the developer are the setup and stable operation of the system, which he chose because of what the community recommended. Everyone chooses for themselves what to use, the main thing that it meets the needs of the developer.

In the future, working on optimization is working with the project code. The developer should understand how Node.js works and what its weaknesses and strengths are. The best thing for Big Data is to develop an architecture that will only be responsible for its part of the work and nothing else. The project should log its result and be able to test the functionality in closed sandboxes. It is important to understand that Node.js offers the developer different ways of solving these problems, so the best solution will be to follow the best practices for developing high-end systems. In particular, the code should handle complex queue operations, cache data, and write the most optimized database queries.

Adhering to the simplest architecture and implementing only Node.js project development best practices, the IoT platform has every chance of working quickly and steadily with any amount of data.

 

IoT analytics in real-time

Typically, the primary purpose of an IoT platform is to collect analytics of hundreds or even tens of thousands of sensors or other devices and share this information with users. In today's world, everything should work in real-time and have a convenient interface for processing this data. Analytics development is a big challenge for a developer who balances speed with ease of implementation. Many people think that it is better to use the ready solution and adjust it to your project. If the project does not have very specific functionality, then this is generally the right approach because one way or another, someone has encountered this problem and is already likely to have successfully solved it. On the other hand, the developer has a number of tasks that are specific, and the project is not quite ordinary.

If the task of developing analytics in real-time, and the developer is very limited in time, it is better to shift this task to services, at least AWS or the like. There are also plenty of ready-made open source solutions on the Internet that allow you to quickly integrate the module into your system and use it.

Of course, a developer can immediately connect a web client to an IoT and simply output data, but there are a number of dangerous nuances. It is important to understand what you tell the user, how he or she should be authorized, and whether the system can control the users and their level of access. The best solution is to split the IoT from WebSocket on a server that will filter the data flow and not be able to directly affect the IoT. Not least is the fact that when developing IoT transmission through WebSocket you need to be careful of memory leak and not complicate the operations of preparing and sending data for analytics.

We know that all operations in the JavaScript browser are related to the capabilities of the device on which the browser is running, so if the data flow is too large, there is a risk of freezing or stopping the browser altogether. This is not a problem for experienced developers, as they are very skilled in using queues and caching and correctly filtering data flow. The developer should target the audience that will be the main focus of this project and optimize first for the needs of the users. Most developers implement SPA (Single Page Application) web applications based on popular frameworks or libraries, such as Angular or React.