Whiz Tools

Snowflake ID Generator

Optional: Unix timestamp in milliseconds (defaults to current time)

Snowflake ID Generator

Introduction

A Snowflake ID is a unique identifier used in distributed systems, originally developed by Twitter. This tool allows you to generate and analyze Snowflake IDs, which are 64-bit integers composed of a timestamp, machine ID, and sequence number.

How Snowflake IDs Work

Snowflake IDs are 64-bit integers structured as follows:

  • 41 bits: Timestamp (milliseconds since a custom epoch)
  • 10 bits: Machine ID (5 bits for data center ID, 5 bits for worker ID)
  • 12 bits: Sequence number

This structure allows for the generation of approximately 4,096 unique IDs per millisecond per machine.

Using the Snowflake ID Generator

  1. (Optional) Set a custom epoch (default is Twitter's epoch: 2010-11-04T01:42:54.657Z)
  2. Enter a machine ID (0-31) and data center ID (0-31)
  3. Click "Generate" to create a new Snowflake ID
  4. The generated ID and its components will be displayed

To parse an existing Snowflake ID, enter it in the "Parse ID" field and click "Parse".

Formula

The Snowflake ID is constructed using bitwise operations:

ID = (timestamp << 22) | (datacenterId << 17) | (workerId << 12) | sequence

Where:

  • timestamp is the number of milliseconds since the epoch
  • datacenterId is a 5-bit integer (0-31)
  • workerId is a 5-bit integer (0-31)
  • sequence is a 12-bit integer (0-4095)

Calculation

The Snowflake ID generator performs the following steps:

  1. Get the current timestamp in milliseconds
  2. Ensure the timestamp is greater than the last used timestamp (for uniqueness)
  3. If the timestamp is the same as the last one, increment the sequence number
  4. If the sequence number overflows (reaches 4096), wait for the next millisecond
  5. Combine the components using bitwise operations to create the final ID

Use Cases

Snowflake IDs are particularly useful in:

  1. Distributed Systems: Generate unique IDs across multiple machines without coordination
  2. High-Volume Data: Create sortable IDs for large datasets
  3. Microservices: Ensure unique identifiers across different services
  4. Database Sharding: Use the timestamp or machine ID component for efficient sharding

Alternatives

While Snowflake IDs are powerful, other ID generation systems include:

  1. UUID (Universally Unique Identifier): Useful when distributed generation is needed without sortability
  2. Auto-incrementing database IDs: Simple but limited to single database instances
  3. ULID (Universally Unique Lexicographically Sortable Identifier): Similar to Snowflake, but with a different structure

Edge Cases and Limitations

  1. Clock Synchronization: Snowflake IDs rely on system time. If the clock moves backwards due to NTP adjustments or daylight saving time changes, it can cause issues with ID generation.

  2. Year 2038 Problem: The 41-bit timestamp will overflow in 2079 (assuming the Twitter epoch). Systems using Snowflake IDs should plan for this eventuality.

  3. Machine ID Collisions: In large distributed systems, ensuring unique machine IDs can be challenging and may require additional coordination.

  4. Sequence Overflow: In extremely high-throughput scenarios, it's possible to exhaust the 4096 sequences per millisecond, potentially causing delays.

  5. Non-monotonicity Across Machines: While IDs are monotonically increasing on a single machine, they may not be strictly monotonic across multiple machines.

History

Snowflake IDs were introduced by Twitter in 2010 to address the need for distributed, time-sortable unique identifiers. They have since been adopted and adapted by many other companies and projects.

Examples

Here are implementations of Snowflake ID generators in various languages:

class SnowflakeGenerator {
  constructor(epoch = 1288834974657, datacenterIdBits = 5, workerIdBits = 5, sequenceBits = 12) {
    this.epoch = BigInt(epoch);
    this.datacenterIdBits = datacenterIdBits;
    this.workerIdBits = workerIdBits;
    this.sequenceBits = sequenceBits;
    this.maxDatacenterId = -1n ^ (-1n << BigInt(datacenterIdBits));
    this.maxWorkerId = -1n ^ (-1n << BigInt(workerIdBits));
    this.sequenceMask = -1n ^ (-1n << BigInt(sequenceBits));
    this.workerIdShift = BigInt(sequenceBits);
    this.datacenterIdShift = BigInt(sequenceBits + workerIdBits);
    this.timestampLeftShift = BigInt(sequenceBits + workerIdBits + datacenterIdBits);
    this.sequence = 0n;
    this.lastTimestamp = -1n;
  }

  nextId(datacenterId, workerId) {
    let timestamp = this.currentTimestamp();

    if (timestamp < this.lastTimestamp) {
      throw new Error('Clock moved backwards. Refusing to generate id');
    }

    if (timestamp === this.lastTimestamp) {
      this.sequence = (this.sequence + 1n) & this.sequenceMask;
      if (this.sequence === 0n) {
        timestamp = this.tilNextMillis(this.lastTimestamp);
      }
    } else {
      this.sequence = 0n;
    }

    this.lastTimestamp = timestamp;

    return ((timestamp - this.epoch) << this.timestampLeftShift) |
           (BigInt(datacenterId) << this.datacenterIdShift) |
           (BigInt(workerId) << this.workerIdShift) |
           this.sequence;
  }

  tilNextMillis(lastTimestamp) {
    let timestamp = this.currentTimestamp();
    while (timestamp <= lastTimestamp) {
      timestamp = this.currentTimestamp();
    }
    return timestamp;
  }

  currentTimestamp() {
    return BigInt(Date.now());
  }
}

// Usage
const generator = new SnowflakeGenerator();
const id = generator.nextId(1, 1);
console.log(`Generated Snowflake ID: ${id}`);
import time
import threading

class SnowflakeGenerator:
    def __init__(self, datacenter_id, worker_id, sequence=0):
        self.datacenter_id = datacenter_id
        self.worker_id = worker_id
        self.sequence = sequence

        self.last_timestamp = -1
        self.epoch = 1288834974657

        self.datacenter_id_bits = 5
        self.worker_id_bits = 5
        self.sequence_bits = 12

        self.max_datacenter_id = -1 ^ (-1 << self.datacenter_id_bits)
        self.max_worker_id = -1 ^ (-1 << self.worker_id_bits)

        self.worker_id_shift = self.sequence_bits
        self.datacenter_id_shift = self.sequence_bits + self.worker_id_bits
        self.timestamp_left_shift = self.sequence_bits + self.worker_id_bits + self.datacenter_id_bits
        self.sequence_mask = -1 ^ (-1 << self.sequence_bits)

        self._lock = threading.Lock()

    def _til_next_millis(self, last_timestamp):
        timestamp = self._get_timestamp()
        while timestamp <= last_timestamp:
            timestamp = self._get_timestamp()
        return timestamp

    def _get_timestamp(self):
        return int(time.time() * 1000)

    def next_id(self):
        with self._lock:
            timestamp = self._get_timestamp()

            if timestamp < self.last_timestamp:
                raise ValueError("Clock moved backwards. Refusing to generate id")

            if timestamp == self.last_timestamp:
                self.sequence = (self.sequence + 1) & self.sequence_mask
                if self.sequence == 0:
                    timestamp = self._til_next_millis(self.last_timestamp)
            else:
                self.sequence = 0

            self.last_timestamp = timestamp

            return ((timestamp - self.epoch) << self.timestamp_left_shift) | \
                   (self.datacenter_id << self.datacenter_id_shift) | \
                   (self.worker_id << self.worker_id_shift) | \
                   self.sequence

## Usage
generator = SnowflakeGenerator(datacenter_id=1, worker_id=1)
snowflake_id = generator.next_id()
print(f"Generated Snowflake ID: {snowflake_id}")
import java.util.concurrent.locks.Lock;
import java.util.concurrent.locks.ReentrantLock;

public class SnowflakeGenerator {
    private final long epoch;
    private final long datacenterIdBits;
    private final long workerIdBits;
    private final long sequenceBits;
    private final long maxDatacenterId;
    private final long maxWorkerId;
    private final long workerIdShift;
    private final long datacenterIdShift;
    private final long timestampLeftShift;
    private final long sequenceMask;

    private long datacenterId;
    private long workerId;
    private long sequence = 0L;
    private long lastTimestamp = -1L;

    private final Lock lock = new ReentrantLock();

    public SnowflakeGenerator(long datacenterId, long workerId) {
        this.epoch = 1288834974657L;
        this.datacenterIdBits = 5L;
        this.workerIdBits = 5L;
        this.sequenceBits = 12L;

        this.maxDatacenterId = ~(-1L << datacenterIdBits);
        this.maxWorkerId = ~(-1L << workerIdBits);

        this.workerIdShift = sequenceBits;
        this.datacenterIdShift = sequenceBits + workerIdBits;
        this.timestampLeftShift = sequenceBits + workerIdBits + datacenterIdBits;
        this.sequenceMask = ~(-1L << sequenceBits);

        if (datacenterId > maxDatacenterId || datacenterId < 0) {
            throw new IllegalArgumentException("datacenterId can't be greater than maxDatacenterId or less than 0");
        }
        if (workerId > maxWorkerId || workerId < 0) {
            throw new IllegalArgumentException("workerId can't be greater than maxWorkerId or less than 0");
        }
        this.datacenterId = datacenterId;
        this.workerId = workerId;
    }

    public long nextId() {
        lock.lock();
        try {
            long timestamp = timeGen();
            if (timestamp < lastTimestamp) {
                throw new RuntimeException("Clock moved backwards. Refusing to generate id");
            }

            if (lastTimestamp == timestamp) {
                sequence = (sequence + 1) & sequenceMask;
                if (sequence == 0) {
                    timestamp = tilNextMillis(lastTimestamp);
                }
            } else {
                sequence = 0L;
            }

            lastTimestamp = timestamp;

            return ((timestamp - epoch) << timestampLeftShift) |
                   (datacenterId << datacenterIdShift) |
                   (workerId << workerIdShift) |
                   sequence;
        } finally {
            lock.unlock();
        }
    }

    private long tilNextMillis(long lastTimestamp) {
        long timestamp = timeGen();
        while (timestamp <= lastTimestamp) {
            timestamp = timeGen();
        }
        return timestamp;
    }

    private long timeGen() {
        return System.currentTimeMillis();
    }

    public static void main(String[] args) {
        SnowflakeGenerator generator = new SnowflakeGenerator(1, 1);
        long id = generator.nextId();
        System.out.println("Generated Snowflake ID: " + id);
    }
}
require 'time'

class SnowflakeGenerator
  def initialize(datacenter_id, worker_id, sequence = 0)
    @datacenter_id = datacenter_id
    @worker_id = worker_id
    @sequence = sequence
    @last_timestamp = -1
    @epoch = 1288834974657

    @datacenter_id_bits = 5
    @worker_id_bits = 5
    @sequence_bits = 12

    @max_datacenter_id = -1 ^ (-1 << @datacenter_id_bits)
    @max_worker_id = -1 ^ (-1 << @worker_id_bits)

    @worker_id_shift = @sequence_bits
    @datacenter_id_shift = @sequence_bits + @worker_id_bits
    @timestamp_left_shift = @sequence_bits + @worker_id_bits + @datacenter_id_bits
    @sequence_mask = -1 ^ (-1 << @sequence_bits)
  end

  def next_id
    timestamp = (Time.now.to_f * 1000).to_i

    raise 'Clock moved backwards' if timestamp < @last_timestamp

    if timestamp == @last_timestamp
      @sequence = (@sequence + 1) & @sequence_mask
      timestamp = til_next_millis(@last_timestamp) if @sequence == 0
    else
      @sequence = 0
    end

    @last_timestamp = timestamp

    ((timestamp - @epoch) << @timestamp_left_shift) |
      (@datacenter_id << @datacenter_id_shift) |
      (@worker_id << @worker_id_shift) |
      @sequence
  end

  private

  def til_next_millis(last_timestamp)
    timestamp = (Time.now.to_f * 1000).to_i
    timestamp = (Time.now.to_f * 1000).to_i while timestamp <= last_timestamp
    timestamp
  end
end

## Usage
generator = SnowflakeGenerator.new(1, 1)
snowflake_id = generator.next_id
puts "Generated Snowflake ID: #{snowflake_id}"
<?php

class SnowflakeGenerator {
    private $epoch;
    private $datacenterIdBits;
    private $workerIdBits;
    private $sequenceBits;
    private $maxDatacenterId;
    private $maxWorkerId;
    private $workerIdShift;
    private $datacenterIdShift;
    private $timestampLeftShift;
    private $sequenceMask;

    private $datacenterId;
    private $workerId;
    private $sequence = 0;
    private $lastTimestamp = -1;

    public function __construct($datacenterId, $workerId) {
        $this->epoch = 1288834974657;
        $this->datacenterIdBits = 5;
        $this->workerIdBits = 5;
        $this->sequenceBits = 12;

        $this->maxDatacenterId = -1 ^ (-1 << $this->datacenterIdBits);
        $this->maxWorkerId = -1 ^ (-1 << $this->workerIdBits);

        $this->workerIdShift = $this->sequenceBits;
        $this->datacenterIdShift = $this->sequenceBits + $this->workerIdBits;
        $this->timestampLeftShift = $this->sequenceBits + $this->workerIdBits + $this->datacenterIdBits;
        $this->sequenceMask = -1 ^ (-1 << $this->sequenceBits);

        if ($datacenterId > $this->maxDatacenterId || $datacenterId < 0) {
            throw new Exception("datacenterId can't be greater than maxDatacenterId or less than 0");
        }
        if ($workerId > $this->maxWorkerId || $workerId < 0) {
            throw new Exception("workerId can't be greater than maxWorkerId or less than 0");
        }
        $this->datacenterId = $datacenterId;
        $this->workerId = $workerId;
    }

    public function nextId() {
        $timestamp = $this->timeGen();

        if ($timestamp < $this->lastTimestamp) {
            throw new Exception("Clock moved backwards. Refusing to generate id");
        }

        if ($this->lastTimestamp == $timestamp) {
            $this->sequence = ($this->sequence + 1) & $this->sequenceMask;
            if ($this->sequence == 0) {
                $timestamp = $this->tilNextMillis($this->lastTimestamp);
            }
        } else {
            $this->sequence = 0;
        }

        $this->lastTimestamp = $timestamp;

        return (($timestamp - $this->epoch) << $this->timestampLeftShift) |
               ($this->datacenterId << $this->datacenterIdShift) |
               ($this->workerId << $this->workerIdShift) |
               $this->sequence;
    }

    private function tilNextMillis($lastTimestamp) {
        $timestamp = $this->timeGen();
        while ($timestamp <= $lastTimestamp) {
            $timestamp = $this->timeGen();
        }
        return $timestamp;
    }

    private function timeGen() {
        return floor(microtime(true) * 1000);
    }
}

// Usage
$generator = new SnowflakeGenerator(1, 1);
$id = $generator->nextId();
echo "Generated Snowflake ID: " . $id . "\n";
using System;
using System.Threading;

public class SnowflakeGenerator
{
    private readonly long _epoch;
    private readonly int _datacenterIdBits;
    private readonly int _workerIdBits;
    private readonly int _sequenceBits;
    private readonly long _maxDatacenterId;
    private readonly long _maxWorkerId;
    private readonly int _workerIdShift;
    private readonly int _datacenterIdShift;
    private readonly int _timestampLeftShift;
    private readonly long _sequenceMask;

    private readonly long _datacenterId;
    private readonly long _workerId;
    private long _sequence = 0L;
    private long _lastTimestamp = -1L;

    private readonly object _lock = new object();

    public SnowflakeGenerator(long datacenterId, long workerId)
    {
        _epoch = 1288834974657L;
        _datacenterIdBits = 5;
        _workerIdBits = 5;
        _sequenceBits = 12;

        _maxDatacenterId = -1L ^ (-1L << _datacenterIdBits);
        _maxWorkerId = -1L ^ (-1L << _workerIdBits);

        _workerIdShift = _sequenceBits;
        _datacenterIdShift = _sequenceBits + _workerIdBits;
        _timestampLeftShift = _sequenceBits + _workerIdBits + _datacenterIdBits;
        _sequenceMask = -1L ^ (-1L << _sequenceBits);

        if (datacenterId > _maxDatacenterId || datacenterId < 0)
        {
            throw new ArgumentException($"datacenterId can't be greater than {_maxDatacenterId} or less than 0");
        }
        if (workerId > _maxWorkerId || workerId < 0)
        {
            throw new ArgumentException($"workerId can't be greater than {_maxWorkerId} or less than 0");
        }
        _datacenterId = datacenterId;
        _workerId = workerId;
    }

    public long NextId()
    {
        lock (_lock)
        {
            var timestamp = TimeGen();

            if (timestamp < _lastTimestamp)
            {
                throw new Exception("Clock moved backwards. Refusing to generate id");
            }

            if (_lastTimestamp == timestamp)
            {
                _sequence = (_sequence + 1) & _sequenceMask;
                if (_sequence == 0)
                {
                    timestamp = TilNextMillis(_lastTimestamp);
                }
            }
            else
            {
                _sequence = 0L;
            }

            _lastTimestamp = timestamp;

            return ((timestamp - _epoch) << _timestampLeftShift) |
                   (_datacenterId << _datacenterIdShift) |
                   (_workerId << _workerIdShift) |
                   _sequence;
        }
    }

    private long TilNextMillis(long lastTimestamp)
    {
        var timestamp = TimeGen();
        while (timestamp <= lastTimestamp)
        {
            timestamp = TimeGen();
        }
        return timestamp;
    }

    private long TimeGen()
    {
        return DateTimeOffset.UtcNow.ToUnixTimeMilliseconds();
    }
}

// Usage
class Program
{
    static void Main(string[] args)
    {
        var generator = new SnowflakeGenerator(1, 1);
        var id = generator.NextId();
        Console.WriteLine($"Generated Snowflake ID: {id}");
    }
}
package main

import (
	"fmt"
	"sync"
	"time"
)

type SnowflakeGenerator struct {
	epoch             int64
	datacenterIdBits  uint
	workerIdBits      uint
	sequenceBits      uint
	maxDatacenterId   int64
	maxWorkerId       int64
	workerIdShift     uint
	datacenterIdShift uint
	timestampLeftShift uint
	sequenceMask      int64

	datacenterId int64
	workerId     int64
	sequence     int64
	lastTimestamp int64

	lock sync.Mutex
}

func NewSnowflakeGenerator(datacenterId, workerId int64) (*SnowflakeGenerator, error) {
	g := &SnowflakeGenerator{
		epoch:             1288834974657,
		datacenterIdBits:  5,
		workerIdBits:      5,
		sequenceBits:      12,
		lastTimestamp:     -1,
	}

	g.maxDatacenterId = -1 ^ (-1 << g.datacenterIdBits)
	g.maxWorkerId = -1 ^ (-1 << g.workerIdBits)

	g.workerIdShift = g.sequenceBits
	g.datacenterIdShift = g.sequenceBits + g.workerIdBits
	g.timestampLeftShift = g.sequenceBits + g.workerIdBits + g.datacenterIdBits
	g.sequenceMask = -1 ^ (-1 << g.sequenceBits)

	if datacenterId > g.maxDatacenterId || datacenterId < 0 {
		return nil, fmt.Errorf("datacenterId can't be greater than %d or less than 0", g.maxDatacenterId)
	}
	if workerId > g.maxWorkerId || workerId < 0 {
		return nil, fmt.Errorf("workerId can't be greater than %d or less than 0", g.maxWorkerId)
	}
	g.datacenterId = datacenterId
	g.workerId = workerId

	return g, nil
}

func (g *SnowflakeGenerator) NextId() (int64, error) {
	g.lock.Lock()
	defer g.lock.Unlock()

	timestamp := g.timeGen()

	if timestamp < g.lastTimestamp {
		return 0, fmt.Errorf("clock moved backwards, refusing to generate id")
	}

	if g.lastTimestamp == timestamp {
		g.sequence = (g.sequence + 1) & g.sequenceMask
		if g.sequence == 0 {
			timestamp = g.tilNextMillis(g.lastTimestamp)
		}
	} else {
		g.sequence = 0
	}

	g.lastTimestamp = timestamp

	return ((timestamp - g.epoch) << g.timestampLeftShift) |
		(g.datacenterId << g.datacenterIdShift) |
		(g.workerId << g.workerIdShift) |
		g.sequence, nil
}

func (g *SnowflakeGenerator) tilNextMillis(lastTimestamp int64) int64 {
	timestamp := g.timeGen()
	for timestamp <= lastTimestamp {
		timestamp = g.timeGen()
	}
	return timestamp
}

func (g *SnowflakeGenerator) timeGen() int64 {
	return time.Now().UnixNano() / int64(time.Millisecond)
}

func main() {
	generator, err := NewSnowflakeGenerator(1, 1)
	if err != nil {
		fmt.Printf("Error creating generator: %v\n", err)
		return
	}

	id, err := generator.NextId()
	if err != nil {
		fmt.Printf("Error generating ID: %v\n", err)
		return
	}

	fmt.Printf("Generated Snowflake ID: %d\n", id)
}

Diagram

Here's a visual representation of the Snowflake ID structure:

Timestamp (41 bits) Machine ID (10 bits) Sequence (12 bits)

64-bit Snowflake ID Structure

References

  1. "Announcing Snowflake." Twitter Engineering Blog, https://blog.twitter.com/engineering/en_us/a/2010/announcing-snowflake
  2. "Snowflake ID." Wikipedia, https://en.wikipedia.org/wiki/Snowflake_ID
  3. "Distributed ID Generation in Microservices." Medium, https://medium.com/swlh/distributed-id-generation-in-microservices-b6ce9a8dd93f
Feedback