Multimodal Embedding & Reranker Models with Sentence Transformers

Sentence Transformers is a Python library for using and training embedding and reranker models for applications like retrieval augmented generation, semantic search, and more. With the v5.4 update, you can now encode and compare texts, images, audio, and videos using the same familiar API. In this blogpost, I'll show you how to use these new multimodal capabilities for both embedding and reranking.

Multimodal embedding models map inputs from different modalities into a shared embedding space, while multimodal reranker models score the relevance of mixed-modality pairs. This opens up use cases like visual document retrieval, cross-modal search, and multimodal RAG pipelines.

Table of Contents

• What are Multimodal Models?
• Installation
• Multimodal Embedding Models
• Loading a Model
• Encoding Images
• Cross-Modal Similarity
• Encoding Queries and Documents
• Multimodal Reranker Models
• Ranking Mixed-Modality Documents
• Predicting Pair Scores
• Retrieve and Rerank
• Input Formats and Configuration
• Supported Input Types
• Checking Modality Support
• Processor and Model kwargs
• Supported Models
• Additional Resources

What are Multimodal Models?

Traditional embedding models convert text into fixed-size vectors. Multimodal embedding models extend this by mapping inputs from different modalities (text, images, audio, or video) into a shared embedding space. This means you can compare a text query against image documents (or vice versa) using the same similarity functions you're already familiar with.

Similarly, traditional reranker (Cross Encoder) models compute relevance scores between pairs of texts. Multimodal rerankers can score pairs where one or both elements are images, combined text-image documents, or other modalities.

For example, you can compare a text query against image documents, find video clips matching a description, or build RAG pipelines that work across modalities.

Installation

Multimodal models require some extra dependencies. Install the extras for the modalities you need (see Installation for more details):

# For image support
pip install -U "sentence-transformers[image]"

# For audio support
pip install -U "sentence-transformers"

# For video support
pip install -U "sentence-transformers"

# Mix and match as needed
pip install -U "sentence-transformers[image,video,train]"

VLM-based models like Qwen3-VL-2B require a GPU with at least ~8 GB of VRAM. For the 8B variants, expect ~20 GB. If you don't have a local GPU, consider using a cloud GPU service or Google Colab. On CPU, these models will be extremely slow; text-only or CLIP models are better suited for CPU inference.

Multimodal Embedding Models

Loading a Model

Loading a multimodal embedding model works exactly like loading a text-only model:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("Qwen/Qwen3-VL-Embedding-2B", revision="refs/pr/23")

The revision argument is required for now because the integration pull requests for these models are still pending. Once they're merged, you'll be able to load them without specifying a revision.

The model automatically detects which modalities it supports, so there's nothing extra to configure. See Processor and Model kwargs if you want to control things like image resolution or model precision.

Encoding Images

With a multimodal model loaded, model.encode() accepts images alongside text. Images can be provided as URLs, local file paths, or PIL Image objects (see Supported Input Types for all accepted formats):

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("Qwen/Qwen3-VL-Embedding-2B", revision="refs/pr/23")

# Encode images from URLs
img_embeddings = model.encode([
 "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg",
 "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg",
])
print(img_embeddings.shape)
# (2, 2048)

Cross-Modal Similarity

You can compute similarities between text embeddings and image embeddings, since the model maps both into the same space:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("Qwen/Qwen3-VL-Embedding-2B", revision="refs/pr/23")

# Encode images
img_embeddings = model.encode([
 "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg",
 "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg",
])

# Encode text queries (one matching + one hard negative per image)
text_embeddings = model.encode([
 "A green car parked in front of a yellow building",
 "A red car driving on a highway",
 "A bee on a pink flower",
 "A wasp on a wooden table",
])

# Compute cross-modal similarities
similarities = model.similarity(text_embeddings, img_embeddings)
print(similarities)
# tensor([[0.5115, 0.1078],
# [0.1999, 0.1108],
# [0.1255, 0.6749],
# [0.1283, 0.2704]])

As expected, "A green car parked in front of a yellow building" is most similar to the car image (0.51), and "A bee on a pink flower" is most similar to the bee image (0.67). The hard negatives ("A red car driving on a highway", "A wasp on a wooden table") correctly receive lower scores.

You might notice that even the best matching scores (0.51, 0.67) aren't very close to 1.0. This is due to the modality gap: embeddings from different modalities tend to cluster in separate regions of the space. Cross-modal similarities are typically lower than within-modal ones (e.g., text-to-text), but the relative ordering is preserved, so retrieval still works well.

Encoding Queries and Documents

For retrieval tasks, encode_query() and encode_document() are the recommended methods. Many retrieval models prepend different instruction prompts depending on whether the input is a query or a document, similar to how chat models might apply different system prompts depending on the goal. Model authors can specify their prompts in the model config, and encode_query() / encode_document() automatically load and apply the correct one:

• encode_query() uses the model's "query" prompt (if available) and sets task="query".
• encode_document() uses the first available prompt from "document", "passage", or "corpus", and sets task="document".

Under the hood, both are thin wrappers around encode(), they just handle prompt selection for you. Here's what cross-modal retrieval looks like:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("Qwen/Qwen3-VL-Embedding-2B", revision="refs/pr/23")

# Encode text queries with the query prompt
query_embeddings = model.encode_query([
 "Find me a photo of a vehicle parked near a building",
 "Show me an image of a pollinating insect",
])

# Encode document screenshots with the document prompt
doc_embeddings = model.encode_document([
 "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg",
 "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg",
])

# Compute similarities
similarities = model.similarity(query_embeddings, doc_embeddings)
print(similarities)
# tensor([[0.3907, 0.1490],
# [0.1235, 0.4872]])

These methods accept the same input types as encode() (images, URLs, multimodal dicts, etc.) and pass through the same parameters. For models without specialized query/document prompts, they behave identically to encode().

Multimodal Reranker Models

Multimodal reranker (CrossEncoder) models score the relevance between pairs of inputs, where each element can be text, an image, audio, video, or a combination. They tend to outperform embedding models in terms of quality, but are slower since they process each pair individually. The currently available pretrained multimodal rerankers focus on text and image inputs, but the architecture supports any modality that the underlying model can handle.

Ranking Mixed-Modality Documents

The rank() method scores and ranks a list of documents against a query, supporting mixed modalities:

from sentence_transformers import CrossEncoder

model = CrossEncoder("Qwen/Qwen3-VL-Reranker-2B", revision="refs/pr/11")

query = "A green car parked in front of a yellow building"
documents = [
 # Image documents (URL or local file path)
 "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg",
 "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg",
 # Text document
 "A vintage Volkswagen Beetle painted in bright green sits in a driveway.",
 # Combined text + image document
 {
 "text": "A car in a European city",
 "image": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg",
 },
]

rankings = model.rank(query, documents)
for rank in rankings:
 print(f"{rank['score']:.4f}\t(document {rank['corpus_id']})")
"""
0.9375 (document 0)
0.5000 (document 3)
-1.2500 (document 2)
-2.4375 (document 1)
"""

The reranker correctly identifies the car image (document 0) as the most relevant result, followed by the combined text+image document about a car in a European city (document 3). The bee image (document 1) scores lowest.

Keep in mind that the modality gap can influence absolute scores: text-image pair scores may occupy a different range than text-text or image-image pair scores.

You can also check which modalities a reranker supports using modalities and supports(), just like with embedding models:

print(model.modalities)
# ['text', 'image', 'video', 'message']

print(model.supports("image"))
# True

# Check if the model supports a specific pair of modalities
print(model.supports(("image", "text")))
# True

Predicting Pair Scores

You can also use predict() to get raw relevance scores for specific pairs of inputs:

from sentence_transformers import CrossEncoder

model = CrossEncoder("jinaai/jina-reranker-m0", trust_remote_code=True)

scores = model.predict([
 ("A green car", "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg"),
 ("A bee on a flower", "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg"),
 ("A green car", "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg"),
])
print(scores)
# [0.9389156 0.96922314 0.46063158]

Retrieve and Rerank

A common pattern is to use an embedding model for fast initial retrieval, then refine the top results with a reranker:

from sentence_transformers import SentenceTransformer, CrossEncoder

# Step 1: Retrieve with an embedding model
embedder = SentenceTransformer("Qwen/Qwen3-VL-Embedding-2B", revision="refs/pr/23")

query = "revenue growth chart"
query_embedding = embedder.encode_query(query)

# Pre-compute corpus embeddings (do this once, then store them)
document_screenshots = [
 "path/to/doc1.png",
 "path/to/doc2.png",
 # ... potentially millions of document screenshots
]
corpus_embeddings = embedder.encode_document(document_screenshots, show_progress_bar=True)

# Simple cosine similarity retrieval, viable as long as embeddings fit in memory
similarities = embedder.similarity(query_embedding, corpus_embeddings)
top_k_indices = similarities.argsort(descending=True)[0][:10]

# Step 2: Rerank the top-k results with a reranker model
reranker = CrossEncoder("nvidia/llama-nemotron-rerank-vl-1b-v2", trust_remote_code=True)

top_k_documents = [document_screenshots[i] for i in top_k_indices]
rankings = reranker.rank(query, top_k_documents)
for rank in rankings:
 print(f"{rank['score']:.4f}\t{top_k_documents[rank['corpus_id']]}")

Since the corpus embeddings are pre-computed, the initial retrieval is fast even over millions of documents. The reranker then provides more accurate scoring over the smaller candidate set.

Input Formats and Configuration

Supported Input Types

Multimodal models accept a variety of input formats. Here's a summary of what you can pass to model.encode():

Checking Modality Support

You can check which modalities a model supports using the modalities property and supports() method:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("Qwen/Qwen3-VL-Embedding-2B", revision="refs/pr/23")

# List all supported modalities
print(model.modalities)
# ['text', 'image', 'video', 'message']

# Check for a specific modality
print(model.supports("image"))
# True
print(model.supports("audio"))
# False

The "message" modality indicates that the model accepts chat-style message inputs with interleaved content. In practice, you rarely need to use this directly. When you pass strings, URLs, or multimodal dicts, the model converts them to the appropriate message format internally. Sentence Transformers supports two message formats:

1. Structured (most VLMs, e.g. Qwen3-VL): Content is a list of typed dicts, e.g. [{"type": "text", "text": "..."}, {"type": "image", "image": ...}]
2. Flat (e.g. Deepseek-V3): Content is a direct value, e.g. "some text"

The format is auto-detected from the model's chat template.

Since all inputs get converted into the same message format internally, you can mix input types in a single encode() call:

embeddings = model.encode([
 # A text input
 "A green car parked in front of a yellow building",
 # An image input (URL)
 "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg",
 # A combined text + image input
 {
 "text": "A car in a European city",
 "image": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg",
 },
])

Processor and Model kwargs

You may want to control image resolution bounds or model precision. Use processor_kwargs and model_kwargs when loading the model:

model = SentenceTransformer(
 "Qwen/Qwen3-VL-Embedding-2B",
 model_kwargs={"attn_implementation": "flash_attention_2", "torch_dtype": "bfloat16"},
 processor_kwargs={"min_pixels": 28 * 28, "max_pixels": 600 * 600},
 revision="refs/pr/23",
)

processor_kwargs controls how inputs are preprocessed (e.g., image resolution bounds). Higher max_pixels means higher quality but more memory and compute. These are

The post Multimodal Embedding & Reranker Models with Sentence Transformers first appeared on Hard Wired.

When you use an app, whether it's a web app or a desktop app, that communicates with a remote server over the internet, you can be 99% sure there is an API server involved. API servers are the hidden part of the application where almost all the magic happens. One option for creating APIs is NestJS, and we will explore it a little bit.

Key Benefits of NestJS:

• Scalability: NestJS is built with scalability in mind, making it easy to manage large-scale applications. Its modular architecture allows developers to split their applications into small, reusable modules.
• TypeScript Support: As a framework built on top of TypeScript, NestJS offers all the benefits of TypeScript, including type safety, improved code readability, and maintainability.
• Extensive Ecosystem: NestJS has a rich ecosystem of libraries and tools, which simplifies the development process. It integrates seamlessly with other popular libraries and frameworks like Express, Fastify, and more.
• Ease of Use: The framework is designed to be developer-friendly, with a well-documented API and a robust CLI that helps in generating boilerplate code, thereby reducing development time.
• Dependency Injection: NestJS uses a powerful dependency injection system, which makes it easier to manage dependencies and promotes better code organization and testability.

Disclaimer: This article is not primarily intended for people who are new to programming. It is written with the expectation of some experience in programming. To be honest, my motivation for writing this article is to compile my NestJS developer notes, and I've chosen to present them in article form. It will not explain NestJS in detail, nor its architecture. Instead, it will focus on solving specific problems I have encountered. I hope you find it useful.

In Chapter 1, we will install NestJS and bring the API server to life.

Make sure you are using the latest LTS version of NodeJS.

For project creation and management, we will use @nestjs/cli. The easiest way is to install it globally.

npm i -g @nestjs/cli

Then, in your projects folder, simply use the nest command to create a new project.

nest new hardwired-nestjs-api-chapter-01

You will be asked which package manager you want to use. I use npm.

When it is done, just delete the .spec.ts files from the src folder. We don't care about tests right now. A bunch of things are already set up, like prettier, eslint, and TypeScript configuration. Useful scripts are prepared in package.json.

"scripts": {
    "build": "nest build",
    "format": "prettier --write \"src/**/*.ts\" \"test/**/*.ts\"",
    "start": "nest start",
    "start:dev": "nest start --watch",
    "start:debug": "nest start --debug --watch",
    "start:prod": "node dist/main",
    "lint": "eslint \"{src,apps,libs,test}/**/*.ts\" --fix",
    "test": "jest",
    "test:watch": "jest --watch",
    "test:cov": "jest --coverage",
    "test:debug": "node --inspect-brk -r tsconfig-paths/register -r ts-node/register node_modules/.bin/jest --runInBand",
    "test:e2e": "jest --config ./test/jest-e2e.json"
  },

For now, we only need start:dev to run the development version. In the src folder, you can find main.ts, which is the entry point to our API, along with app.controller.ts, app.module.ts, and app.service.ts. You can find a perfect explanation of the NestJS architecture in the chapters Controllers, Providers, and Modules in the NestJS documentation.

main.ts is pretty simple. Just includes @netjs/core, app.module and bootstrap our application.

import { NestFactory } from '@nestjs/core';
import { AppModule } from './app.module';

async function bootstrap() {
  const app = await NestFactory.create(AppModule);
  await app.listen(3000);
}

bootstrap();

When you run npm run start:dev, the API will run on port 3000, and you will be able to access it at http://localhost:3000.

Controllers are where you define your routes. In app.controller.ts, there is one GET route.

import { Controller, Get } from '@nestjs/common';
import { AppService } from './app.service';

@Controller()
export class AppController {
  constructor(private readonly appService: AppService) {}

  @Get()
  getHello(): string {
    return this.appService.getHello();
  }
}

The @Get decorator without parameters in app.controller.ts indicates that you will access this route with /. It's the root route, and you can access it at http://localhost:3000. This route calls the service method getHello(), which returns the string Hello World!. Services are where your business logic resides.

import { Injectable } from '@nestjs/common';

@Injectable()
export class AppService {
  getHello(): string {
    return 'Hello World!';
  }
}

You can access http://localhost:3000 in your browser, and if the development server is running, you will see Hello World!. However, as an API developer, you may want a better way to call your API during development. I recommend using Insomnia.

When you call http://localhost:3000 using Insomnia, you will get the same result as in the browser. However, as you add more routes and JSON responses, you will appreciate the usefulness of Insomnia.

Hello World!

Hello World is okay, but not very useful. Let's build something more meaningful.

First, we should discuss how to organize our routes to avoid a mess later. A good way is to organize them into resources. A resource can be user, article, comment, etc. For resource bases, we will use plurals, for example, users, articles, and comments. It is a good practice to follow REST principles when creating an API. One of my favorite articles on this topic is Best Practices for Designing a Pragmatic RESTful API.

Sometimes it is easier for programmers to use just GET and POST methods for all routes, but this does not adhere to RESTful principles. RESTful principles provide strategies to handle CRUD actions. It is better to use the full potential of HTTP methods. For example, when we take the resource user:

• GET /users - retrieves a list of users
• GET /users/1 - retrieves the user with id 1
• POST /users - creates a new user
• PUT /users/1 - updates the user with id 1
• PATCH /users/1 - partially updates the user with id 1
• DELETE /users/1 - deletes the user with id 1

You can also combine resources. For instance, to get the articles for a user, one route could be GET /users/1/articles.

If you want to like an article, the API endpoint could be PUT /articles/1/like. You can use aliases, filtering, sorting, limiting, and pagination constructs, but this is a complex and highly opinionated field suitable for many articles. I think the best quick dive is provided in the article I mentioned earlier, written by Vinay Sahni.

Earlier in the article, I said "Let's build something more meaningful," but that was a lie. For a quick demonstration of how to create a resource and follow methods, we will use something completely useless.

We will create a route GET /zodiac-signs. This route will return a list of zodiac signs with date ranges. I borrowed the list from here.

To create a NestJS resource, we will use the nest cli.

nest g res zodiac-signs --no-spec

• nest - nest command
• g - generate
• res - resource
• zodiac-sign - resource-name
• --no-spec - do not generate tests

After you enter the command, you will be asked what type of transport layer you want to use. Choose REST API. The next question will be whether you would like to create CRUD entry points. I choose no; I will create them on my own.

When it is done, in your src folder, you will see a folder named zodiac-signs, and it will contain zodiac-signs.controller.ts, zodiac-signs.module.ts, and zodiac-signs.service.ts.

Nest CLI automatically connects your new resource/module to the app.

We will start with zodiac-signs.service.ts.

import { Injectable } from '@nestjs/common';

@Injectable()
export class ZodiacSignsService {
    signs = {
        aries: {
            symbol: '♈',
            dateMin: '2000-03-21',
            dateMax: '2000-04-20',
        },
        taurus: {
            symbol: '♉',
            dateMin: '2000-04-21',
            dateMax: '2000-05-21',
        },
        gemini: {
            symbol: '♊',
            dateMin: '2000-05-22',
            dateMax: '2000-06-21',
        },
        cancer: {
            symbol: '♋',
            dateMin: '2000-06-22',
            dateMax: '2000-07-22',
        },
        leo: {
            symbol: '♌',
            dateMin: '2000-07-23',
            dateMax: '2000-08-22',
        },
        virgo: {
            symbol: '♍',
            dateMin: '2000-08-23',
            dateMax: '2000-09-23',
        },
        libra: {
            symbol: '♎',
            dateMin: '2000-09-24',
            dateMax: '2000-10-23',
        },
        scorpio: {
            symbol: '♏',
            dateMin: '2000-10-24',
            dateMax: '2000-11-22',
        },
        sagittarius: {
            symbol: '♐',
            dateMin: '2000-11-23',
            dateMax: '2000-12-21',
        },
        capricorn: {
            symbol: '♑',
            dateMin: '2000-12-22',
            dateMax: '2000-01-20',
        },
        aquarius: {
            symbol: '♒',
            dateMin: '2000-01-21',
            dateMax: '2000-02-19',
        },
        pisces: {
            symbol: '♓',
            dateMin: '2000-02-20',
            dateMax: '2000-03-20',
        },
    };

    findAll() {
        return this.signs;
    }
}

In the ZodiacSignsService class, we have our logic. The list of signs is stored as a class property named signs. The method that returns this list is findAll(). It is a good practice for the route GET /zodiac-signs to use the service method findAll, and for GET /zodiac-signs/:name to use the service method findOne. This uniform naming system will help you orient yourself in a large code base.

Next will be zodiac-signs.controller.ts.

import { Controller, Get } from '@nestjs/common';
import { ZodiacSignsService } from './zodiac-signs.service';

@Controller('zodiac-signs')
export class ZodiacSignsController {
  constructor(private readonly zodiacSignsService: ZodiacSignsService) {}

  @Get()
  findAll() {
    return this.zodiacSignsService.findAll();
  }
}

We create a method findAll decorated with @Get decorator, which, without parameters, instructs the app router to point to the findAll method in the ZodiacSignsController in response to a GET /zodiac-signs request.

That's all for now. Make sure the development server is running (npm run start:dev). Then, you can access http://localhost:3000/zodiac-signs. The expected result is the same JSON as we have in ZodiacSignsService.

Okay, it's not much, but it's enough to get started. In the next chapter, we'll take a look at DTOs (Data Transfer Objects).

You can find project on my GitHub.

Happy coding!

The post NestJS Chapter 01 – New Project first appeared on Hard Wired.

Sometimes you want to create your own npm package. Here is the way to set up an npm package in TypeScript.

In your project directory, initialize the npm object.

npm init

Install development dependencies.

npm install --save-dev typescript ts-node

ts-node enables you to directly execute TypeScript on Node.js. See more on the package website.

Set up the TypeScript project.

npx tsc --init

In the generated tsconfig.json, set outDir to dist.

...
"outDir": "./dist", /* Specify an output folder for all emitted files. */
...

In the root directory, create the src folder and in the folder index.ts which will be the entry point in your package.

mkdir src
touch ./src/index.ts

Create a .gitignore file.

/node_modules
# Ignore test-related files
/coverage.data
/coverage/
# Build files
/dist

Install the tsup package. It is a bundler and is powered by esbuild.

In the context of JavaScript, a bundler is a tool that takes multiple JavaScript files and their dependencies and combines them into a single file or a few files, which can then be included in a web application. The main purpose of a bundler is to manage and optimize the delivery of code to the browser, enhancing performance and simplifying development.

npm install --save-dev tsup

In the root directory, create the tsup.config.ts file.

import { defineConfig } from "tsup";
export default defineConfig({
entry: ["src/index.ts"],
format: ["cjs", "esm"], // Build for commonJS and ESmodules
dts: true, // Generate declaration file (.d.ts)
splitting: false,
sourcemap: true,
clean: true,
});

Update these properties in package.json

...
"main": "./dist/index.js",
"module": "./dist/index.mjs",
"types": "./dist/index.d.ts",
"files": [
    "dist"
],
"scripts": {
    "build": "tsup"
},
...

In your index.ts, create a trivial function you will export in this package.

export function add(a: number, b: number): number {
  return a + b;
}

Build your package with npm run build. In the dist directory, you will see the build of your project.

TADA! Now you have a base project for your npm package.

The post How to create TypeScript NPM package project first appeared on Hard Wired.

TypeScript compared to JavaScript is a little bit more complex. If you do server-side JavaScript, you just call node your-script.js and it runs. When you want to use TypeScript, you need to install a few things and do a few steps to make your code runnable.

Make sure you have installed the latest version of nodejs on your system.

Create your project folder, change the current working directory, and install typescript as your development dependency.

npm i -D typescript

Initialize your nodejs project.

npm init

Then use npx to initialize a new typescript project.

npx tsc --init

A bunch of files should be in your project:

• package.json
• package-lock.json
• tsconfig.json

Edit tsconfig.json, find outDir, and change it if you do not want compiled files in your root directory. Change it to the build folder.

...
outDir": "./build",   /* Specify an output folder for all emitted files. */
...

Now it's time to create your first TypeScript code.

Create a file index.ts and just put some random code.

async function main() {
    const i: number = 1;

    console.log('Random stuff', i);
}

main()

Save it and then you can do a manual compilation.

npx tsc

This compiles your project and the output will be stored in the build folder. You will find the file index.js there. Do a test run with node ./build/index.js.

In the console, you should see:

Random stuff 1

You can use npx tsc -w to activate watch mode. When you change anything in your project, it will be recompiled automatically.

To make it a little bit easier, you can edit the scripts property in package.json to merge compilation and run into one command.

...
"scripts": {
        "start": "npx tsc && node ./build/index.js"
},
...

When you call npm run start, your project will be compiled and run.

During code development, you should use a linter to check your code using certain rules to avoid code inconsistency. An easy way is to use Google TypeScript Style.

Install gts as a development dependency.

npm i -D gts

Then initialize gts with npx.

npx gts init

It will ask you some questions. If your typescript version is higher than suggested, just say no. Then you will be asked about updating tsconfig.json, say yes.

Some big changes will be made.

The src folder will be created with an index.ts file in it. Remove your old index.ts from the root folder.

In package.json, update your start script.

...
"start": "npx tsc && node ./build/src/index.js",
...

Make sure you have eslint support activated in your editor

You will get an index.ts example file in the src directory. It is all wrong. If you have eslint support activated in your editor, it should scream at you with lots of errors.

console.log("Try npm run lint/fix!");

const longString = 'Lorem ipsum dolor sit amet, consectetur adipiscing elit. Integer ut aliquet diam.';

const trailing = 'Semicolon'

            const why={am:'I tabbed?'};

const iWish = "I didn't have a trailing space..."; 

const sicilian = true;;

const vizzini = (!!sicilian) ? !!!sicilian : sicilian;

const re = /foo   bar/;

export function doSomeStuff(withThis: string, andThat: string, andThose: string[]) {
    //function on one line
    if(!Boolean(andThose.length)) {return false;}
    console.log(withThis);
    console.log(andThat);
    console.dir(andThose);
    console.log(longString, trailing, why, iWish, vizzini, re);
    return;
}
// TODO: more examples

Just run npm run fix to let gts fix your code. You will get this:

console.log('Try npm run lint/fix!');

const longString =
  'Lorem ipsum dolor sit amet, consectetur adipiscing elit. Integer ut aliquet diam.';

const trailing = 'Semicolon';

const why = {am: 'I tabbed?'};

const iWish = "I didn't have a trailing space...";

const sicilian = true;

const vizzini = sicilian ? !sicilian : sicilian;

const re = /foo {3}bar/;

export function doSomeStuff(
  withThis: string,
  andThat: string,
  andThose: string[]
) {
  //function on one line
  if (!andThose.length) {
    return false;
  }
  console.log(withThis);
  console.log(andThat);
  console.dir(andThose);
  console.log(longString, trailing, why, iWish, vizzini, re);
  return;
}
// TODO: more examples

If it is working, you have just created your typescript project with a linter. It is a good thing to configure your code editor to perform npm run fix with every save.

Enjoy TypeScript coding.

The post How to create a new TypeScript project first appeared on Hard Wired.

Pokud potřebujete na ESP32 spouštět naplánované úlohy, určitě se nemůžete spoléhat na vnitřní časování. Budete potřebovat nějaký externí modul reálného času (RTC), který vám na vyžádání vrátí přesný čas.

V bastlířské oblasti se hodně často používá DS1307. Vyniká hlavně v tom, že je extrémně levný. Na Aliexpressu se dá koupit za necelých 14 CZK. Cena kompenzuje určitou nepřesnost. Udává se zpoždění cca 5 minut za měsíc.

Sice několika násobně dražší, ale pořád směšně levný je DS3231. Na Aliexpressu se dá pořídit za 60 CZK.

U Lásky 128 CZK plus poštovné.

Tento modul je daleko přesnější a obsahuje tepelně kompenzovaný krystalový oscilátor (TCXO), který odolává změnám teplot. Pro náš testovací projekt použijeme DS3231.

DS3231

Modul umí sledovat roky, měsíce, dny, hodiny, minuty a sekundy. Umí dokonce přestupné roky do roku 2100. Dokáže pracovat v 12 hodinovém nebo 24 hodinovém formátu. Obsahuje dva programovatelné alarmy.

Modul má piny pro připojení přes sběrnici I2C. Má také pin INT, který umí produkovat signál přerušení a pin SQW pro generování "square wave" signálu na frekvencích 1Hz, 4kHz, 8kHz nebo 32kHz.

Přesnost modulu je +-2 minuty za rok.

Modul se napájí pomocí knoflíkové baterie CR2032, LIR2032 nebo ekvivalentní o průměru 20mm. Při použití 220mAh baterie je teoretická životnost 8 let.

Modul obsahuje 32 bytový AT24C32 EEPROM paměťový čip s milionem zapisovacích cyklů. Neslouží pro běh hodin, může být použit pro zápis logů nebo libovolných dat.

Vyčerpávající popis modulu najdete na lastminuteengineers.com.

Zapojení

Pomocí I2C připojíme k ESP modulu.

Program

Pokud nejsou hodiny nastaveny nebo mají starší čas, než je čas při kompilaci kódu, přenastaví se na aktuální čas. Po inicializaci modulu se vytvoří naplánovaná úloha pomocí modulu TaskScheduler a každých 10 sekund vypíše aktuální čas modulu do sériové konzole. Zbytek v komentovaném kódu.

main.cpp

/**
 * @file main.cpp
 * @author Hard Wired
 * @brief ESP32 I2C RTC test project.
 * @details
 *      Based on https://github.com/Makuna/Rtc/blob/master/examples/DS3231_Simple/DS3231_Simple.ino example.
 *      Modul tutorial https://lastminuteengineers.com/ds3231-rtc-arduino-tutorial/
 * @version 0.1
 * @date 2023-02-08
 */

#include <Arduino.h>

#include <Wire.h>
#include <RtcDS3231.h>
#include <TaskScheduler.h>

const int SECOND_IN_MILS = 1000;
const int TEN_SECONDS_IN_MILS = SECOND_IN_MILS * 10;

/**
 * @brief Vytvoření plánovače
 *
 */
Scheduler runner;

/**
 * @brief Vytvoření RTC objektu a připojení pomocí I2C sběrnice.
 *
 * @return RtcDS3231<TwoWire>
 */
RtcDS3231<TwoWire> Rtc(Wire);

/**
 * @brief Vytáhne z objektu "Rtc" poslední chybový stav a vypíše chybovou hlášku¨
 *        a dá vědět pomocí true/false.
 * @details see https://www.arduino.cc/reference/en/language/functions/communication/wire/endtransmission/
 *
 * @param errorTopic
 * @return true
 * @return false
 */
bool wasError(const char *errorTopic = "")
{
    uint8_t error = Rtc.LastError();
    if (error != 0)
    {
        Serial.print("[");
        Serial.print(errorTopic);
        Serial.print("] WIRE communications error (");
        Serial.print(error);
        Serial.print(") : ");

        switch (error)
        {
        case Rtc_Wire_Error_None:
            Serial.println("(none?!)");
            break;
        case Rtc_Wire_Error_TxBufferOverflow:
            Serial.println("transmit buffer overflow");
            break;
        case Rtc_Wire_Error_NoAddressableDevice:
            Serial.println("no device responded");
            break;
        case Rtc_Wire_Error_UnsupportedRequest:
            Serial.println("device doesn't support request");
            break;
        case Rtc_Wire_Error_Unspecific:
            Serial.println("unspecified error");
            break;
        case Rtc_Wire_Error_CommunicationTimeout:
            Serial.println("communications timed out");
            break;
        }
        return true;
    }
    return false;
}

/**
 * @brief Vypíše naformátované datum do sériové konzole.
 *
 * @param dt
 */
void printDateTime(const RtcDateTime &dt)
{
    char buffer[20];

    sprintf(buffer, "%d-%02d-%02d %02d:%02d:%02d", dt.Year(), dt.Month(), dt.Day(), dt.Hour(), dt.Minute(), dt.Second());

    Serial.println(buffer);
}

void printDateTimeAndTemperature()
{
    if (!Rtc.IsDateTimeValid())
    {
        if (!wasError("loop IsDateTimeValid"))
        {
            // Common Causes:
            //    1) the battery on the device is low or even missing and the power line was disconnected
            Serial.println("RTC lost confidence in the DateTime!");
        }
    }

    RtcDateTime now = Rtc.GetDateTime();
    if (!wasError("loop GetDateTime"))
    {
        printDateTime(now);
    }

    RtcTemperature temp = Rtc.GetTemperature();
    if (!wasError("loop GetTemperature"))
    {
        temp.Print(Serial);
        // you may also get the temperature as a float and print it
        // Serial.print(temp.AsFloatDegC());
        Serial.println("C");
    }
}

/**
 * @brief Vytvoření naplánované úlohy. Poběží navždy a spustí se každých 10 sekund.
 *
 * @return Task
 */
Task PrintDateTimeAndTemperatureTask(TEN_SECONDS_IN_MILS, TASK_FOREVER, &printDateTimeAndTemperature);

void setup()
{
    Serial.begin(115200);

    Serial.print("compiled: ");
    // makro které expanduje datum při kompilaci kódu na "mmm dd yyyy" řetězec
    Serial.print(__DATE__);
    Serial.print(' ');
    // makro které expanduje čas při kompilaci kódu na "hh:MM:ss" řetězec
    Serial.println(__TIME__);

    Rtc.Begin();

#if defined(WIRE_HAS_TIMEOUT)
    Wire.setWireTimeout(3000 /* us */, true /* reset_on_timeout */);
#endif

    /**
     * @brief Vytvoří objekt RtcDateTime a nastaví ho podle času kompilace kódu.
     *
     */
    RtcDateTime compiled = RtcDateTime(__DATE__, __TIME__);
    printDateTime(compiled);
    Serial.println();

    /**
     * @brief Pokud neni RTC nastaveno nastaví datum a čas.
     *
     */
    if (!Rtc.IsDateTimeValid())
    {
        if (!wasError("setup IsDateTimeValid"))
        {
            // Common Causes:
            //    1) first time you ran and the device wasn't running yet
            //    2) the battery on the device is low or even missing

            Serial.println("RTC lost confidence in the DateTime!");

            // following line sets the RTC to the date & time this sketch was compiled
            // it will also reset the valid flag internally unless the Rtc device is
            // having an issue

            Rtc.SetDateTime(compiled);
        }
    }

    /**
     * @brief V případě že není RTC spuštěno spustíme ho.
     *
     */
    if (!Rtc.GetIsRunning())
    {
        if (!wasError("setup GetIsRunning"))
        {
            Serial.println("RTC was not actively running, starting now");
            Rtc.SetIsRunning(true);
        }
    }

    /**
     * @brief V případě, že je čas hodin starší než čas kompilace, nastaví se nový čas. Vypíše do sériové konzole informaci o nastavení RTC.
     *
     */
    RtcDateTime now = Rtc.GetDateTime();
    if (!wasError("setup GetDateTime"))
    {
        if (now < compiled)
        {
            Serial.println("RTC is older than compile time, updating DateTime");
            Rtc.SetDateTime(compiled);
        }
        else if (now > compiled)
        {
            Serial.println("RTC is newer than compile time, this is expected");
        }
        else if (now == compiled)
        {
            Serial.println("RTC is the same as compile time, while not expected all is still fine");
        }
    }

    /*
     * Nastaví RTC modul do požadovaného stavu.
     */
    Rtc.Enable32kHzPin(false);
    wasError("setup Enable32kHzPin");
    // The INT/SQW pin on the DS3231 provides either an interrupt signal (due to alarm conditions) or a nice square wave at 1Hz, 4kHz, 8kHz, or 32kHz.
    Rtc.SetSquareWavePin(DS3231SquareWavePin_ModeNone);
    wasError("setup SetSquareWavePin");

    /**
     * @brief Přiřazení naplánované úlohy do plánovače.
     *
     */
    runner.addTask(PrintDateTimeAndTemperatureTask);

    /**
     * @brief Povolení naplánované úlohy.
     *
     */
    PrintDateTimeAndTemperatureTask.enable();
}

void loop()
{
    runner.execute(); // spouštění plánovače
}

platformio.ini

[env:esp32dev]
platform = espressif32
board = esp32dev
framework = arduino
monitor_speed = 115200

lib_deps =
  makuna/RTC @ ^2.3.6
  arkhipenko/TaskScheduler @ ^3.7.0

The post ESP32 RTC DS3231 first appeared on Hard Wired.