Sing's Log

Knowledge worth sharing


  • Home

  • Tags

  • Timeline

  • Portfolio

  • Search

Remove duplicate documents from Mongodb by keys

Posted on 2022-11-17 | Edited on 2023-01-16 |

Background

To ensure there is no duplicate documents, we could create an unique index.
However, it will throw error when there are duplicates entries in the collection:

MongoDB cannot create a unique index on the specified index field(s) if the collection already contains data that would violate the unique constraint for the index.

Remove the duplicates by keys

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
const collection = db.collection('MyCollection');
const operations: Promise<any>[] = [];
console.time('aggregation');

await collection
.aggregate([
{
$group: {
_id: {
key1: '$key1',
key2: '$key2',
},
dups: {
$push: '$_id',
},
count: {
$sum: 1,
},
},
},
{
$match: {
_id: {
$ne: null,
},
count: {
$gt: 1,
},
},
},
])
.forEach((doc) => {
console.log(doc);
doc.dups.slice(1).forEach((duplicateId: string) => {
operations.push(collection.deleteOne({ _id: duplicateId }));
});
});

console.timeEnd('aggregation');

console.time('remove duplicate');
await Promise.all(operations);
console.timeEnd('remove duplicate');

Create the unique index

1
await collection.createIndex({ key1: 1, key2: 1 }, { unique: true, name: 'unique_index' });

Mongodb connection test inside a container

Posted on 2022-07-28 | Edited on 2023-01-16 |

Background

To test the connection between a container and the mongoDB.

Steps

  1. Install mongo client

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    # Install required package
    apt-get install gnupg

    # import the MongoDB public GPG Key
    wget -qO - https://www.mongodb.org/static/pgp/server-6.0.asc | apt-key add -

    # Create a /etc/apt/sources.list.d/mongodb-org-6.0.list file for MongoDB
    echo "deb http://repo.mongodb.org/apt/debian buster/mongodb-org/6.0 main" | tee /etc/apt/sources.list.d/mongodb-org-6.0.list

    # Reload local package database.
    apt-get update

    # Install the MongoDB packages.
    apt-get install -y mongodb-org
  2. Test the connection

    1
    2
    3
    4
    5
    # Login to mongo (Use mongo or mongosh, depends on the mongodb package version)
    mongosh "mongodb://username:[email protected]:5/dbname?authSource=admin"

    # Check the DB name
    db

Reference

  • https://www.mongodb.com/docs/manual/tutorial/install-mongodb-on-debian/#install-mongodb-community-edition

Build Puppeteer Docker image to run inside K8s cluster

Posted on 2022-07-27 | Edited on 2023-01-16 |

Background

  • To run Puppeteer, you can’t just use an official Nodejs image and npm install puppeteer
  • Below is the Dockerfile that I have tried and proved as a working solution
  • More information see the Puppeteer official troubleshooting

Dockerfile

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
FROM alpine

# Installs latest Chromium (100) package.
RUN apk add --no-cache \
chromium \
nss \
freetype \
harfbuzz \
ca-certificates \
ttf-freefont \
nodejs \
yarn

# Tell Puppeteer to skip installing Chrome. We'll be using the installed package.
ENV PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=true \
PUPPETEER_EXECUTABLE_PATH=/usr/bin/chromium-browser

COPY package.json yarn.lock ./
RUN yarn

# Puppeteer v13.5.0 works with Chromium 100.
# RUN yarn add [email protected]

# Add user so we don't need --no-sandbox.
RUN addgroup -S pptruser && adduser -S -G pptruser pptruser \
&& mkdir -p /home/pptruser/Downloads /app \
&& chown -R pptruser:pptruser /home/pptruser \
&& chown -R pptruser:pptruser /app

# Run everything after as non-privileged user.
USER pptruser

WORKDIR /app

COPY . .

EXPOSE 3003
CMD ["yarn", "start"]

Enlarge Click Area by CSS pseudo element

Posted on 2022-06-06 | Edited on 2023-01-16 |

An utility class .enlarge-click-area to enlarge the click area of any elements.

Plain CSS

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
.enlarge-click-area {
position: relative;
}
.enlarge-click-area::after {
content: "";
display: block;
position: absolute;
left: 0;
top: 0;
width: 160%;
height: 160%;
/* How to calculate 18.75%:
Extra width for left-hand side: (160% - 100%) / 2 = 30%
Transform percentage will relative to the width: 30% / 160% = 18.75%
*/
transform: translate(-18.75%, -18.75%);
}

SCSS

1
2
3
4
5
6
7
8
9
10
11
12
13
14
=enlargeClickArea($ratio: 2)
position: relative

&:after
$offset: percentage((1 - $ratio) / 2 / $ratio)

content: ''
display: block
position: absolute
left: 0
top: 0
width: percentage($ratio)
height: percentage($ratio)
transform: translate($offset, $offset)

Generate Let's Encrypt certificate manually by DNS challenge

Posted on 2022-05-28 | Edited on 2023-01-16 |

Steps

  1. Create a folder for the certificate files

    1
    mkdir /tmp/cert
  2. Use the certbot command with docker:

    1
    docker run -v /tmp/cert:/etc/letsencrypt/archive -it certbot/certbot certonly --preferred-challenges dns --manual
  3. Answer the questions

  4. Go to your DNS provider to add the TXT records specified in the challenge

  5. Before hitting enter, ensure your record has published by dig tool

  6. Hit enter then you will get the certificates under /tmp/cert/{yourdomain} in your Host machine

Reference

  • Let’s encrypt - How it works?

Coding faster with Vim + IDE

Posted on 2021-09-24 | Edited on 2023-01-16 |

Vim X IDE

  • My slides share
  • Vim tutorial

JetBrains IDE + vim

  • Install ideavim
    • Custom setting
      1. Click V icon on the right bottom corner
      2. Open and edit ~/.ideavimrc
  • Tennis Kata demo - Dart + Android Studio

VScode + Vim

  • Install vscodevim
  • Setup instruction
  • Tennis Kata demo - Vscode + Javascript

SSH to server without password

Posted on 2021-09-15 | Edited on 2023-01-16 |

The requirement to ssh to server without password

To ssh without password, you must meet the condition:

  • Private key in client ~/.ssh/
  • Public key in server ~/.ssh/authorized_keys

Steps to setup

  1. Create ssh key pairs by ssh-keygen

    1
    2
    3
    4
    5
    6
    7
    # Generate key
    ssh-keygen

    # Check the generated key pair
    ls ~/.ssh
    -rw------- 1 sing staff 1679 Jun 8 2018 id_rsa
    -rw-r--r-- 1 sing staff 404 Jun 8 2018 id_rsa.pub
  2. Login to your server

  3. Copy the content of the publicKey (id_rsa.pub) to server’s ~/.ssh/authorized_keys

    1
    echo "${public key content}" >> .ssh/authorized_keys
  4. Verify ssh without password

    1
    ssh [email protected]

Device pixel ratio - How to select a proper size image that looks sharp and display fast on different devices

Posted on 2021-09-14 | Edited on 2023-01-16 |

What is Device Pixel Ratio?

The term device pixel ratio plays the crucial role in different languages’ styling. The definition of Device pixel ratio from MDN:

The devicePixelRatio of Window interface returns the ratio of the resolution in physical pixels to the resolution in CSS pixels for the current display device.

My translation:

1
Device pixel ratio = (Device physical pixel width) / (Width of a device in a program)

For instance an iPhone12:

  • Physical Width: 1170px
  • CSS width: 390px
  • Device Ratio Pixel: 1170px / 390px = 3

How to select a proper width image

1
Proper image size = devicePixelRatio * width

If you setup a image with style width:150px, the phone will use 150 * 3 = 450 physical pixel to render, so the proper width of the image should be 450

Common Device Pixel Ratio

Reference

  • https://www.mydevice.io/
  • https://developer.mozilla.org/en-US/docs/Web/API/Window/devicePixelRatio

Display iphone/android screen on Mac

Posted on 2020-11-16 | Edited on 2023-01-16 |

Display iphone screen on Mac

  1. Connect your iphone through wire.
  2. Open QuickTime Player
  3. File > New Movie Recording
  4. In the recording screen, select the dropdown > camera > your iphone name

Display Android Device on Mac

  1. Install scrcpy by brew install scrcpy
  2. If you don’t have adb, run brew cask install android-platform-tools
  3. Connect your Android phone through wire
  4. excute scrcpy

Schedule backup Mongodb to AWS S3 with DroneCI cronjob

Posted on 2019-10-18 | Edited on 2023-01-16 |

To schedule backup mongodb to AWS S3 could be done extremely easy in any docker pipeline, here is an example using DroneCI docker pipeline and cronjob feature. The complete code example could be found here: https://github.com/Asing1001/mongodb-s3-backup.

  1. Create .drone.jsonnet file using docker pipeline
  2. Use mongodump in step1 to create archive
  3. Upload the archive by s3 plugin in step2

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    local backupLocation = 'mongo-archive.gz';

    [{
    kind: 'pipeline',
    name: 'default',
    steps: [
    {
    name: 'mongodump',
    image: 'mongo:3.6.5',
    environment: {
    DB_URI: {
    from_secret: 'db_uri'
    },
    },
    commands: [
    'mongodump --gzip --uri="$DB_URI" --archive=' + backupLocation,
    ],
    },
    {
    name: 'upload',
    image: 'plugins/s3',
    settings: {
    bucket: 'mongo-backup',
    access_key: {
    from_secret: 'aws_access_key_id',
    },
    secret_key: {
    from_secret: 'aws_secret_access_key',
    },
    source: backupLocation,
    target: '/',
    },
    },
    ],
    trigger: {
    event: ['cron', 'push'],
    branch: 'master'
    },
    }]
  4. Activate the project in DroneCI and fill up variables aws_access_key_id, aws_secret_access_key, db_uri

  5. Setup Cron jobs in Drone project’s settings
    drone-mongo

  6. Push the repository and verify the job :)

Reference

  • Drone
  • mongodump
  • drone S3 plugin
12…8
Sing Ming Chen

Sing Ming Chen

Sing's log, a developer's blog

77 posts
193 tags
GitHub E-Mail Linkedin Facebook StackOverflow
© 2023 Sing Ming Chen
Powered by Hexo v3.9.0
|
Theme — NexT.Gemini v6.3.0