Knowledge worth sharing

Introduction

When cloning a project from a remote source, it’s crucial to synchronize the Terraform states from the backend to maintain consistency and accuracy.

Syncing Remote Backend State: A Quick Guide

Follow these steps to efficiently sync your remote backend state:

1. Initialize and Install Dependencies

First, initiate the process and update dependencies using the following command:

1
2
3
4

terraform init

# If you need to upgrade dependencies
terraform init --upgrade

2. Streamline Variable Input

Instead of manually inputting or exporting variables each time, simplify the process by creating a terraform.tfvars file:

1

echo "foo=bar" > terraform.tfvars

3. Removing Items Not Present in Remote

In cases where items are removed from the remote but haven’t been reflected in the state, use the following command:

1

terraform state rm <resource>.<name>

4. Importing New Items to Sync State

If new items have been added but aren’t yet synced with the state, consult the provider’s documentation to confirm import support. Then use the provided syntax:

1

terraform import <resource>.<name> <id>

Conclusion

Properly synchronizing Terraform backend states ensures your project’s integrity and consistency, especially when working with remote repositories. Following these concise steps will streamline your workflow and enhance your overall development process.

In this blog post, we will explore how to train a machine learning model to solve CAPTCHA audio challenges. CAPTCHA is a widely used security measure on the web to prevent automated bots from accessing websites. It presents users with various tests to determine if they are human, one of which is the audio challenge. By training a model to solve these audio challenges, we can automate the process and improve user experience. Let’s dive into the code and understand how it works.

The Setup

Before we get into the code, let’s understand the directory structure and dependencies required for this project. Here’s an overview:

1. login-captcha-example: This directory contains the audio files used for training and testing the model.
2. material: This directory contains the labeled training audio files. Each audio file represents a digit in the captcha.
3. svm_model.pkl: This is the trained SVM model file that will be generated during training.
4. temp: This directory will be used to store temporary audio files generated during the prediction process.

To get started, make sure you have the following dependencies installed:

• os: A module for interacting with the operating system.
• librosa: A library for audio and music signal analysis.
• numpy: A library for mathematical operations on multi-dimensional arrays.
• sklearn.svm: The SVM (Support Vector Machine) model implementation from scikit-learn.
• sklearn.metrics: A module for evaluating the performance of machine learning models.
• subprocess: A module for spawning new processes and executing system commands.
• string.Template: A class for string templating.
• joblib: A library for serializing Python objects to disk and loading them back into memory.

Code Walkthrough

Background

To ensure there is no duplicate documents, we could create an unique index.
However, it will throw error when there are duplicates entries in the collection:

MongoDB cannot create a unique index on the specified index field(s) if the collection already contains data that would violate the unique constraint for the index.

Remove the duplicates by keys

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43

const collection = db.collection('MyCollection');
const operations: Promise<any>[] = [];
console.time('aggregation');

await collection
  .aggregate([
    {
      $group: {
        _id: {
          key1: '$key1',
          key2: '$key2',
        },
        dups: {
          $push: '$_id',
        },
        count: {
          $sum: 1,
        },
      },
    },
    {
      $match: {
        _id: {
          $ne: null,
        },
        count: {
          $gt: 1,
        },
      },
    },
  ])
  .forEach((doc) => {
    console.log(doc);
    doc.dups.slice(1).forEach((duplicateId: string) => {
      operations.push(collection.deleteOne({ _id: duplicateId }));
    });
  });

console.timeEnd('aggregation');

console.time('remove duplicate');
await Promise.all(operations);
console.timeEnd('remove duplicate');

Create the unique index

1

await collection.createIndex({ key1: 1, key2: 1 }, { unique: true, name: 'unique_index' });

Background

To test the connection between a container and the mongoDB.

Steps

1. Install mongo client

   1 2 3 4 5 6 7 8 9 10 11 12 13 14

   # Install required package apt-get install gnupg # import the MongoDB public GPG Key wget -qO - https://www.mongodb.org/static/pgp/server-6.0.asc | apt-key add - # Create a /etc/apt/sources.list.d/mongodb-org-6.0.list file for MongoDB echo "deb http://repo.mongodb.org/apt/debian buster/mongodb-org/6.0 main" | tee /etc/apt/sources.list.d/mongodb-org-6.0.list # Reload local package database. apt-get update # Install the MongoDB packages. apt-get install -y mongodb-org

2. Test the connection

   1 2 3 4 5

   # Login to mongo (Use mongo or mongosh, depends on the mongodb package version) mongosh "mongodb://username:[email protected]:5/dbname?authSource=admin" # Check the DB name db

Reference

• https://www.mongodb.com/docs/manual/tutorial/install-mongodb-on-debian/#install-mongodb-community-edition

Background

• To run Puppeteer, you can’t just use an official Nodejs image and npm install puppeteer
• Below is the Dockerfile that I have tried and proved as a working solution
• More information see the Puppeteer official troubleshooting

Dockerfile

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

FROM alpine

# Installs latest Chromium (100) package.
RUN apk add --no-cache \
  chromium \
  nss \
  freetype \
  harfbuzz \
  ca-certificates \
  ttf-freefont \
  nodejs \
  yarn

# Tell Puppeteer to skip installing Chrome. We'll be using the installed package.
ENV PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=true \
  PUPPETEER_EXECUTABLE_PATH=/usr/bin/chromium-browser

COPY package.json yarn.lock ./
RUN yarn

# Puppeteer v13.5.0 works with Chromium 100.
# RUN yarn add [email protected]

# Add user so we don't need --no-sandbox.
RUN addgroup -S pptruser && adduser -S -G pptruser pptruser \
  && mkdir -p /home/pptruser/Downloads /app \
  && chown -R pptruser:pptruser /home/pptruser \
  && chown -R pptruser:pptruser /app

# Run everything after as non-privileged user.
USER pptruser

WORKDIR /app

COPY . .

EXPOSE 3003
CMD ["yarn", "start"]

An utility class .enlarge-click-area to enlarge the click area of any elements.

Plain CSS

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

.enlarge-click-area {
  position: relative;
}
.enlarge-click-area::after {
  content: "";
  display: block;
  position: absolute;
  left: 0;
  top: 0;
  width: 160%;
  height: 160%;
  /* How to calculate 18.75%:
    Extra width for left-hand side: (160% - 100%) / 2 = 30%
    Transform percentage will relative to the width: 30% / 160% = 18.75%
  */
  transform: translate(-18.75%, -18.75%);
}

SCSS

1
2
3
4
5
6
7
8
9
10
11
12
13
14

=enlargeClickArea($ratio: 2)
  position: relative

  &:after
    $offset: percentage((1 - $ratio) / 2 / $ratio)

    content: ''
    display: block
    position: absolute
    left: 0
    top: 0
    width: percentage($ratio)
    height: percentage($ratio)
    transform: translate($offset, $offset)

Steps

1. Create a folder for the certificate files

   1	mkdir /tmp/cert

2. Use the certbot command with docker:

   1	docker run -v /tmp/cert:/etc/letsencrypt/archive -it certbot/certbot certonly --preferred-challenges dns --manual

3. Answer the questions

   

4. Go to your DNS provider to add the TXT records specified in the challenge

5. Before hitting enter, ensure your record has published by dig tool

   

6. Hit enter then you will get the certificates under /tmp/cert/{yourdomain} in your Host machine

   

Reference

• Let’s encrypt - How it works?

Vim X IDE

• My slides share
• Vim tutorial

JetBrains IDE + vim

• Install ideavim
  • Custom setting
    1. Click V icon on the right bottom corner
    2. Open and edit ~/.ideavimrc
• Tennis Kata demo - Dart + Android Studio

VScode + Vim

• Install vscodevim
• Setup instruction
• Tennis Kata demo - Vscode + Javascript

The requirement to ssh to server without password

To ssh without password, you must meet the condition:

• Private key in client ~/.ssh/
• Public key in server ~/.ssh/authorized_keys

Steps to setup

1. Create ssh key pairs by ssh-keygen

   1 2 3 4 5 6 7

   # Generate key ssh-keygen # Check the generated key pair ls ~/.ssh -rw------- 1 sing staff 1679 Jun 8 2018 id_rsa -rw-r--r-- 1 sing staff 404 Jun 8 2018 id_rsa.pub

2. Login to your server

3. Copy the content of the publicKey (id_rsa.pub) to server’s ~/.ssh/authorized_keys

   1	echo "${public key content}" >> ~/.ssh/authorized_keys

4. Make sure the sshd setting /etc/ssh/sshd_config allow publickey authentication,

   1	PubkeyAuthentication yes

   Restart by sudo systemctl restart ssh if you modify the sshd setting.

5. Verify ssh without password

   1	ssh username@serverip

What is Device Pixel Ratio?

The term device pixel ratio plays the crucial role in different languages’ styling. The definition of Device pixel ratio from MDN:

The devicePixelRatio of Window interface returns the ratio of the resolution in physical pixels to the resolution in CSS pixels for the current display device.

My translation:

1

Device pixel ratio = (Device physical pixel width) / (Width of a device in a program)

For instance an iPhone12:

• Physical Width: 1170px
• CSS width: 390px
• Device Ratio Pixel: 1170px / 390px = 3

How to select a proper width image

1

Proper image size = devicePixelRatio * width

If you setup a image with style width:150px, the phone will use 150 * 3 = 450 physical pixel to render, so the proper width of the image should be 450

Common Device Pixel Ratio

Reference

• https://www.mydevice.io/
• https://developer.mozilla.org/en-US/docs/Web/API/Window/devicePixelRatio