When you use an app, whether it's a web app or a desktop app, that communicates with a remote server over the internet, you can be 99% sure there is an API server involved. API servers are the hidden part of the application where almost all the magic happens. One option for creating APIs is NestJS, and we will explore it a little bit.

Key Benefits of NestJS:

• Scalability: NestJS is built with scalability in mind, making it easy to manage large-scale applications. Its modular architecture allows developers to split their applications into small, reusable modules.
• TypeScript Support: As a framework built on top of TypeScript, NestJS offers all the benefits of TypeScript, including type safety, improved code readability, and maintainability.
• Extensive Ecosystem: NestJS has a rich ecosystem of libraries and tools, which simplifies the development process. It integrates seamlessly with other popular libraries and frameworks like Express, Fastify, and more.
• Ease of Use: The framework is designed to be developer-friendly, with a well-documented API and a robust CLI that helps in generating boilerplate code, thereby reducing development time.
• Dependency Injection: NestJS uses a powerful dependency injection system, which makes it easier to manage dependencies and promotes better code organization and testability.

Disclaimer: This article is not primarily intended for people who are new to programming. It is written with the expectation of some experience in programming. To be honest, my motivation for writing this article is to compile my NestJS developer notes, and I've chosen to present them in article form. It will not explain NestJS in detail, nor its architecture. Instead, it will focus on solving specific problems I have encountered. I hope you find it useful.

In Chapter 1, we will install NestJS and bring the API server to life.

Make sure you are using the latest LTS version of NodeJS.

For project creation and management, we will use @nestjs/cli. The easiest way is to install it globally.

npm i -g @nestjs/cli

Then, in your projects folder, simply use the nest command to create a new project.

nest new hardwired-nestjs-api-chapter-01

You will be asked which package manager you want to use. I use npm.

When it is done, just delete the .spec.ts files from the src folder. We don't care about tests right now. A bunch of things are already set up, like prettier, eslint, and TypeScript configuration. Useful scripts are prepared in package.json.

"scripts": {
    "build": "nest build",
    "format": "prettier --write \"src/**/*.ts\" \"test/**/*.ts\"",
    "start": "nest start",
    "start:dev": "nest start --watch",
    "start:debug": "nest start --debug --watch",
    "start:prod": "node dist/main",
    "lint": "eslint \"{src,apps,libs,test}/**/*.ts\" --fix",
    "test": "jest",
    "test:watch": "jest --watch",
    "test:cov": "jest --coverage",
    "test:debug": "node --inspect-brk -r tsconfig-paths/register -r ts-node/register node_modules/.bin/jest --runInBand",
    "test:e2e": "jest --config ./test/jest-e2e.json"
  },

For now, we only need start:dev to run the development version. In the src folder, you can find main.ts, which is the entry point to our API, along with app.controller.ts, app.module.ts, and app.service.ts. You can find a perfect explanation of the NestJS architecture in the chapters Controllers, Providers, and Modules in the NestJS documentation.

main.ts is pretty simple. Just includes @netjs/core, app.module and bootstrap our application.

import { NestFactory } from '@nestjs/core';
import { AppModule } from './app.module';

async function bootstrap() {
  const app = await NestFactory.create(AppModule);
  await app.listen(3000);
}

bootstrap();

When you run npm run start:dev, the API will run on port 3000, and you will be able to access it at http://localhost:3000.

Controllers are where you define your routes. In app.controller.ts, there is one GET route.

import { Controller, Get } from '@nestjs/common';
import { AppService } from './app.service';

@Controller()
export class AppController {
  constructor(private readonly appService: AppService) {}

  @Get()
  getHello(): string {
    return this.appService.getHello();
  }
}

The @Get decorator without parameters in app.controller.ts indicates that you will access this route with /. It's the root route, and you can access it at http://localhost:3000. This route calls the service method getHello(), which returns the string Hello World!. Services are where your business logic resides.

import { Injectable } from '@nestjs/common';

@Injectable()
export class AppService {
  getHello(): string {
    return 'Hello World!';
  }
}

You can access http://localhost:3000 in your browser, and if the development server is running, you will see Hello World!. However, as an API developer, you may want a better way to call your API during development. I recommend using Insomnia.

When you call http://localhost:3000 using Insomnia, you will get the same result as in the browser. However, as you add more routes and JSON responses, you will appreciate the usefulness of Insomnia.

Hello World!

Hello World is okay, but not very useful. Let's build something more meaningful.

First, we should discuss how to organize our routes to avoid a mess later. A good way is to organize them into resources. A resource can be user, article, comment, etc. For resource bases, we will use plurals, for example, users, articles, and comments. It is a good practice to follow REST principles when creating an API. One of my favorite articles on this topic is Best Practices for Designing a Pragmatic RESTful API.

Sometimes it is easier for programmers to use just GET and POST methods for all routes, but this does not adhere to RESTful principles. RESTful principles provide strategies to handle CRUD actions. It is better to use the full potential of HTTP methods. For example, when we take the resource user:

• GET /users - retrieves a list of users
• GET /users/1 - retrieves the user with id 1
• POST /users - creates a new user
• PUT /users/1 - updates the user with id 1
• PATCH /users/1 - partially updates the user with id 1
• DELETE /users/1 - deletes the user with id 1

You can also combine resources. For instance, to get the articles for a user, one route could be GET /users/1/articles.

If you want to like an article, the API endpoint could be PUT /articles/1/like. You can use aliases, filtering, sorting, limiting, and pagination constructs, but this is a complex and highly opinionated field suitable for many articles. I think the best quick dive is provided in the article I mentioned earlier, written by Vinay Sahni.

Earlier in the article, I said "Let's build something more meaningful," but that was a lie. For a quick demonstration of how to create a resource and follow methods, we will use something completely useless.

We will create a route GET /zodiac-signs. This route will return a list of zodiac signs with date ranges. I borrowed the list from here.

To create a NestJS resource, we will use the nest cli.

nest g res zodiac-signs --no-spec

• nest - nest command
• g - generate
• res - resource
• zodiac-sign - resource-name
• --no-spec - do not generate tests

After you enter the command, you will be asked what type of transport layer you want to use. Choose REST API. The next question will be whether you would like to create CRUD entry points. I choose no; I will create them on my own.

When it is done, in your src folder, you will see a folder named zodiac-signs, and it will contain zodiac-signs.controller.ts, zodiac-signs.module.ts, and zodiac-signs.service.ts.

Nest CLI automatically connects your new resource/module to the app.

We will start with zodiac-signs.service.ts.

import { Injectable } from '@nestjs/common';

@Injectable()
export class ZodiacSignsService {
    signs = {
        aries: {
            symbol: '♈',
            dateMin: '2000-03-21',
            dateMax: '2000-04-20',
        },
        taurus: {
            symbol: '♉',
            dateMin: '2000-04-21',
            dateMax: '2000-05-21',
        },
        gemini: {
            symbol: '♊',
            dateMin: '2000-05-22',
            dateMax: '2000-06-21',
        },
        cancer: {
            symbol: '♋',
            dateMin: '2000-06-22',
            dateMax: '2000-07-22',
        },
        leo: {
            symbol: '♌',
            dateMin: '2000-07-23',
            dateMax: '2000-08-22',
        },
        virgo: {
            symbol: '♍',
            dateMin: '2000-08-23',
            dateMax: '2000-09-23',
        },
        libra: {
            symbol: '♎',
            dateMin: '2000-09-24',
            dateMax: '2000-10-23',
        },
        scorpio: {
            symbol: '♏',
            dateMin: '2000-10-24',
            dateMax: '2000-11-22',
        },
        sagittarius: {
            symbol: '♐',
            dateMin: '2000-11-23',
            dateMax: '2000-12-21',
        },
        capricorn: {
            symbol: '♑',
            dateMin: '2000-12-22',
            dateMax: '2000-01-20',
        },
        aquarius: {
            symbol: '♒',
            dateMin: '2000-01-21',
            dateMax: '2000-02-19',
        },
        pisces: {
            symbol: '♓',
            dateMin: '2000-02-20',
            dateMax: '2000-03-20',
        },
    };

    findAll() {
        return this.signs;
    }
}

In the ZodiacSignsService class, we have our logic. The list of signs is stored as a class property named signs. The method that returns this list is findAll(). It is a good practice for the route GET /zodiac-signs to use the service method findAll, and for GET /zodiac-signs/:name to use the service method findOne. This uniform naming system will help you orient yourself in a large code base.

Next will be zodiac-signs.controller.ts.

import { Controller, Get } from '@nestjs/common';
import { ZodiacSignsService } from './zodiac-signs.service';

@Controller('zodiac-signs')
export class ZodiacSignsController {
  constructor(private readonly zodiacSignsService: ZodiacSignsService) {}

  @Get()
  findAll() {
    return this.zodiacSignsService.findAll();
  }
}

We create a method findAll decorated with @Get decorator, which, without parameters, instructs the app router to point to the findAll method in the ZodiacSignsController in response to a GET /zodiac-signs request.

That's all for now. Make sure the development server is running (npm run start:dev). Then, you can access http://localhost:3000/zodiac-signs. The expected result is the same JSON as we have in ZodiacSignsService.

Okay, it's not much, but it's enough to get started. In the next chapter, we'll take a look at DTOs (Data Transfer Objects).

You can find project on my GitHub.

Happy coding!

The post NestJS Chapter 01 – New Project first appeared on Hard Wired.

Sometimes you want to create your own npm package. Here is the way to set up an npm package in TypeScript.

In your project directory, initialize the npm object.

npm init

Install development dependencies.

npm install --save-dev typescript ts-node

ts-node enables you to directly execute TypeScript on Node.js. See more on the package website.

Set up the TypeScript project.

npx tsc --init

In the generated tsconfig.json, set outDir to dist.

...
"outDir": "./dist", /* Specify an output folder for all emitted files. */
...

In the root directory, create the src folder and in the folder index.ts which will be the entry point in your package.

mkdir src
touch ./src/index.ts

Create a .gitignore file.

/node_modules
# Ignore test-related files
/coverage.data
/coverage/
# Build files
/dist

Install the tsup package. It is a bundler and is powered by esbuild.

In the context of JavaScript, a bundler is a tool that takes multiple JavaScript files and their dependencies and combines them into a single file or a few files, which can then be included in a web application. The main purpose of a bundler is to manage and optimize the delivery of code to the browser, enhancing performance and simplifying development.

npm install --save-dev tsup

In the root directory, create the tsup.config.ts file.

import { defineConfig } from "tsup";
export default defineConfig({
entry: ["src/index.ts"],
format: ["cjs", "esm"], // Build for commonJS and ESmodules
dts: true, // Generate declaration file (.d.ts)
splitting: false,
sourcemap: true,
clean: true,
});

Update these properties in package.json

...
"main": "./dist/index.js",
"module": "./dist/index.mjs",
"types": "./dist/index.d.ts",
"files": [
    "dist"
],
"scripts": {
    "build": "tsup"
},
...

In your index.ts, create a trivial function you will export in this package.

export function add(a: number, b: number): number {
  return a + b;
}

Build your package with npm run build. In the dist directory, you will see the build of your project.

TADA! Now you have a base project for your npm package.

The post How to create TypeScript NPM package project first appeared on Hard Wired.

TypeScript compared to JavaScript is a little bit more complex. If you do server-side JavaScript, you just call node your-script.js and it runs. When you want to use TypeScript, you need to install a few things and do a few steps to make your code runnable.

Make sure you have installed the latest version of nodejs on your system.

Create your project folder, change the current working directory, and install typescript as your development dependency.

npm i -D typescript

Initialize your nodejs project.

npm init

Then use npx to initialize a new typescript project.

npx tsc --init

A bunch of files should be in your project:

• package.json
• package-lock.json
• tsconfig.json

Edit tsconfig.json, find outDir, and change it if you do not want compiled files in your root directory. Change it to the build folder.

...
outDir": "./build",   /* Specify an output folder for all emitted files. */
...

Now it's time to create your first TypeScript code.

Create a file index.ts and just put some random code.

async function main() {
    const i: number = 1;

    console.log('Random stuff', i);
}

main()

Save it and then you can do a manual compilation.

npx tsc

This compiles your project and the output will be stored in the build folder. You will find the file index.js there. Do a test run with node ./build/index.js.

In the console, you should see:

Random stuff 1

You can use npx tsc -w to activate watch mode. When you change anything in your project, it will be recompiled automatically.

To make it a little bit easier, you can edit the scripts property in package.json to merge compilation and run into one command.

...
"scripts": {
        "start": "npx tsc && node ./build/index.js"
},
...

When you call npm run start, your project will be compiled and run.

During code development, you should use a linter to check your code using certain rules to avoid code inconsistency. An easy way is to use Google TypeScript Style.

Install gts as a development dependency.

npm i -D gts

Then initialize gts with npx.

npx gts init

It will ask you some questions. If your typescript version is higher than suggested, just say no. Then you will be asked about updating tsconfig.json, say yes.

Some big changes will be made.

The src folder will be created with an index.ts file in it. Remove your old index.ts from the root folder.

In package.json, update your start script.

...
"start": "npx tsc && node ./build/src/index.js",
...

Make sure you have eslint support activated in your editor

You will get an index.ts example file in the src directory. It is all wrong. If you have eslint support activated in your editor, it should scream at you with lots of errors.

console.log("Try npm run lint/fix!");

const longString = 'Lorem ipsum dolor sit amet, consectetur adipiscing elit. Integer ut aliquet diam.';

const trailing = 'Semicolon'

            const why={am:'I tabbed?'};

const iWish = "I didn't have a trailing space..."; 

const sicilian = true;;

const vizzini = (!!sicilian) ? !!!sicilian : sicilian;

const re = /foo   bar/;

export function doSomeStuff(withThis: string, andThat: string, andThose: string[]) {
    //function on one line
    if(!Boolean(andThose.length)) {return false;}
    console.log(withThis);
    console.log(andThat);
    console.dir(andThose);
    console.log(longString, trailing, why, iWish, vizzini, re);
    return;
}
// TODO: more examples

Just run npm run fix to let gts fix your code. You will get this:

console.log('Try npm run lint/fix!');

const longString =
  'Lorem ipsum dolor sit amet, consectetur adipiscing elit. Integer ut aliquet diam.';

const trailing = 'Semicolon';

const why = {am: 'I tabbed?'};

const iWish = "I didn't have a trailing space...";

const sicilian = true;

const vizzini = sicilian ? !sicilian : sicilian;

const re = /foo {3}bar/;

export function doSomeStuff(
  withThis: string,
  andThat: string,
  andThose: string[]
) {
  //function on one line
  if (!andThose.length) {
    return false;
  }
  console.log(withThis);
  console.log(andThat);
  console.dir(andThose);
  console.log(longString, trailing, why, iWish, vizzini, re);
  return;
}
// TODO: more examples

If it is working, you have just created your typescript project with a linter. It is a good thing to configure your code editor to perform npm run fix with every save.

Enjoy TypeScript coding.

The post How to create a new TypeScript project first appeared on Hard Wired.

O co jde

Uděláme bota co načte webovou stránku, klikne na čudlík a stáhne soubor.

Použijeme k tomu knihovnu Puppeteer a samotný kód budeme pouštět v Dockeru.

Větší podrobnosti a vysvětlení zakládání NodeJs projektu v Dockeru najde v článku NodeJS develompent v Dockeru.

Příprava projektu

Založíme nový projekt:

mkdir puppet
cd puppet
npm init -y

Upravíme package.json a přidáme do něj knihovnu puppeteer:

{
  "name": "puppet",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "start": "node index.js"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "dependencies": {
    "puppeteer": "^19.2.0"
  }
}

Vytvoříme Dockerfile, který použije jako základ obraz s node v16, nainstaluje Chrome prohlížeč a nakopíruje projekt do Docker obrazu.

FROM node:16

RUN apt-get update && apt-get install gnupg wget -y && \
  wget --quiet --output-document=- https://dl-ssl.google.com/linux/linux_signing_key.pub | gpg --dearmor > /etc/apt/trusted.gpg.d/google-archive.gpg && \
  sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list' && \
  apt-get update && \
  apt-get install google-chrome-stable -y --no-install-recommends && \
  rm -rf /var/lib/apt/lists/*

ENV PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=true \
    PUPPETEER_EXECUTABLE_PATH=/usr/bin/google-chrome-stable

WORKDIR /app
COPY package*.json .
RUN npm install
COPY . ./

Vytvoříme index.js soubor a necháme ho jen vypsat zprávu do konzole.

touch index.js

console.log('Hello from docker')

Není úplně špatné vytvořit i .dockerignore soubor.

node_modules
Dockerfile
.dockerignore
.git
.gitignore
README.md
docker-compose*

Sestavíme obraz.

docker build -t puppet-app-image .

A spustíme Docker kontejner.

# %cd% je aktuální složka ve Windows, pro jiné systémy použijte adekvátní náhradu

docker run -d --tty --name puppet-app -v %cd%:/app -v /app/node_modules puppet-app-image

--tty parametr je nutný. V opačném případě se Docker kontejner spustí a pak hned úspěšně ukončí. Za normálních okolností předpokládá, že v něm něco poběží a až se to ukončí tak skončí taky. TTY toto chování potlačí.

Jakmile kontejner běží, můžeme se do něj přihlásit:

docker exec -it puppet-app bash

Vevnitř se ocitneme ve složce /app, kde je náš projekt a můžeme zkusit spustit náš program.

nodejs index.js

Měl by nám odpovědět Hello from docker.

Bot

Pro ukázku bude bot stahovat zdrojový kód puppeteeru z githubu. Upravíme index.js následovně.

const puppeteer = require('puppeteer')

;(async () => {
    const browser = await puppeteer.launch({
        headless: true,
        args: ['--no-sandbox'],
    })

    const page = await browser.newPage()
    page.setViewport({ width: 1920, height: 1080 })

    const client = await page.target().createCDPSession()
    await client.send("Page.setDownloadBehavior", {
        behavior: "allow",
        downloadPath: './'
    })

    await page.goto(
        'https://github.com/puppeteer/puppeteer/releases/tag/puppeteer-core-v19.2.0',
        { waitUntil: 'networkidle2' }
    )
    await page.waitForTimeout(1000)

    await page.click('#repo-content-turbo-frame div.mb-3 > details > div > div > ul > li:nth-child(1) > div.d-flex.flex-justify-start.col-12.col-lg-9 > a')
    await page.waitForTimeout(5000)

    await page.screenshot({ path: './screen.jpg', fullPage: true })

    await browser.close()
})()

Naimportujeme do našeho projektu knihovnu puppeteer.

const puppeteer = require('puppeteer')

Budeme volat asynchronní funkce v java scriptu a budeme používat klíčové slovo await, které čeká na provedení asynchronní funkce. Problém je to, že await se může objevit jen v asynchronní funkci. Tudíž náš kód musíme do jedné takové zabalit a rovnou spustit.

;(async () => {
    // code
})()

Vytvoříme nový headless prohlížeč.

const browser = await puppeteer.launch({
    headless: true,
    args: ['--no-sandbox'],
})

Vytvoříme novou stránku/tab a nastavíme rozlišení.

const page = await browser.newPage()
page.setViewport({ width: 1920, height: 1080 })

Pro tuto konkrétní stránku nastavíme automatické stahovaní do určitého adresáře. V našem případě adresář s projektem. Takhle nebudeme vyvolávat dialog pro umístění stahovaného souboru.

const client = await page.target().createCDPSession()
await client.send("Page.setDownloadBehavior", {
    behavior: "allow",
    downloadPath: './'
})

V naší stránce/tabu otevřeme požadovanou github stránku.

await page.goto(
    'https://github.com/puppeteer/puppeteer/releases/tag/puppeteer-core-v19.2.0',
    { waitUntil: 'networkidle2' }
)
await page.waitForTimeout(1000)

Klikneme na odkaz pro stažení a chvíli počkáme, než se soubor stáhne

await page.click('#repo-content-turbo-frame div.mb-3 > details > div > div > ul > li:nth-child(1) > div.d-flex.flex-justify-start.col-12.col-lg-9 > a')
await page.waitForTimeout(5000)

K definování toho na co se má kliknout se používá CSS Selector (draft, mdn). Ten pomocí html elementů, tříd, atributů a dalších dokáže sestavit unikátní cestu k prvku na stránce. Stránky generované nějakým frameworkem umí potrápit i s takhle dlouhým selektorem, jako v tomto případě.

Pak už si jen uděláme momentku, v jakém stavu stránku opouštíme.

await page.screenshot({ path: './screen.jpg', fullPage: true })

A prohlížeč korektně zavřeme.

await browser.close()

Spuštění kódu

V konzoli se připojíme k našemu kontejneru.

docker exec -it puppet-app bash

A spustíme našeho bota.

node index.js

Pokud vše proběhne, jak má, tak ve složce projektu budete mít jak stažený soubor, tak fotku stránky těsně před tím než jste ji opustili.

Toto je velice hrubý nástřel toho, co Puppeteer dokáže na konkrétním případě. Výhodou je, že se opravdu spustí prohlížeč, ve kterém můžeme ovládat i dynamicky generované single page aplikace. Nevýhodou je, že pokud se struktura stránky změní moc, tak musíte upravit i váš kód.

Happy coding!

The post Puppeteer download BOT first appeared on Hard Wired.