• Skip to primary navigation
  • Skip to content

Luc Russell

Data Engineering and Full Stack Development

  • Home
  • Blog

Kafka Quick Reference

11/07/2016 by Luc

  • Create A Topic
  • Delete a Topic
  • Add a Partition
  • Describe a Topic
  • Reassign partitions
  • Checking Lag
  • How to Set Up librdkafka with Pykafka
  • Kafkacat

Create A Topic

1
2
./kafka-topics.sh --create --zookeeper myzookeeper:2181 --replication-factor 3 --partitions 3 --topic mytopic
 

Delete a Topic

1
2
./kafka-topics.sh --zookeeper myzookeeper:2181 --delete --topic mytopic
 

Add a Partition

1
2
./kafka-topics.sh --zookeeper myzookeeper:2181 --alter --topic mytopic --partitions 2
 

Describe a Topic

Note this describes all the topics, regardless of what topic is supplied.

1
2
./kafka-topics.sh --describe mytopic
 

Reassign Partitions

Create a file reassign.json:

1
2
3
{ "version":1, "partitions":
[{"topic":"my-topic", "partition":0, "replicas":[0,1,2] }] }
 

Then execute:

1
2
./kafka-reassign-partitions.sh --zookeeper myzookeeper:2181 --reassignment-json-file reassign.json --execute
 

Checking Lag

A simple solution for quickly checking lag is as follows. However, a longer term solution is to set up a Grafana instance and report lag to this using statsd.

1
2
./kafka-consumer-offset-checker.sh --zookeeper myzookeeper:2181 --group my_group_id --topic mytopic
 

How to Set Up librdkafka with Pykafka

Follow this reference.

In summary:

1
2
3
4
5
6
    git clone https://github.com/edenhill/librdkafka.git
    cd librdkafka/
    ./configure
    make
    sudo make install
 

Then:

1
2
3
4
git clone https://github.com/Parsely/pykafka.git
cd pykafka
python setup.py develop
 

The last part, calling setup.py develop to put pykafka in develop mode, is necessary for creating the link between the two.

Kafkacat

kafkacat is a useful tool, here is a kafkacat quick reference.

Filed Under: quick reference Tagged With: kafka

Docker Quick Reference

10/22/2016 by Luc

Table of Contents

  • Docker Compose Force Rebuild
  • Working With Docker for Mac
    • Aliasing 0.0.0.0 to work around dynamic IPs
    • Accessing The Xyve Virtual Machine
    • Add an Internal Company Registry or Repo
  • Get Into a Docker Container without docker-enter
  • Get Into a Docker Image which Fails to Start
  • Free up Disk Space Used by Docker
  • Temporarily Install a Package for Troubleshooting
  • Run A Container With Env Variables for Testing
  • Remove Containers Created and Exited 2 Weeks ago
  • Remove All Exited Containers
  • Grep Docker stdout and stderr
  • Keep a Container Running
  • Run A Container With a Link to Another
  • Logs
  • Other References

Docker Compose Force Rebuild

Also see here.

1
2
3
4
docker-compose rm -v -f
docker-compose build --no-cache
docker-compose up --force-recreate
 

For compose files where you have trouble removing referenced images, try:

1
2
docker-compose down --rmi all
 

See Docker Compose reference.

Remove a Volume

1
2
3
docker volume ls
docker volume rm -f some_grafana_data
 

You might then see:

1
2
Error response from daemon: unable to remove volume: remove some_grafana_data: volume is in use - [4633083c16d8cab09af2163a2e76105ce1a7e2a5e2d23289f7b67125c607c9a6]
 

In which case:

1
2
3
docker rm -f 4633083c16d8cab09af2163
docker volume rm -f some_grafana_data
 

Working With Docker for Mac

Aliasing 0.0.0.0 to work around dynamic IPs

To solve this problem:

I want to connect from a container to a service on the host. The
mac has a changing IP address or no address if you have no network access

The current recommendation is to attach an unused IP to the lo0 interface on the Mac e.g.

1
2
sudo ifconfig lo0 alias 10.200.10.1/24
 

Make sure your service is listening on this address or 0.0.0.0
(i.e. not 127.0.0.1). Then any containers which need to access the
service can use the 10.200.10.1 address.

Accessing The Xyve Virtual Machine

You might need to do this after restarting/upgrading docker. To get into the xyve vm use screen:

1
2
screen ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty
 

The login is ‘root’, no password.

Add an Internal Company Registry or Repo

1
2
vi /etc/hosts
 

and add the address e.g.:

1
2
10.10.10.100   my-registry.mycompany.com
 

Exit screen with Ctrl-a then k.

Here is a screen reference: http://ss64.com/osx/screen.html.

Get Into a Docker Container without docker-enter

1
2
docker exec -it 963ffd75c1b5 /bin/bash
 

Get Into a Docker Image which Fails to Start

1
2
docker run -it applications_mycontainer /bin/bash
 

Use docker images to find the correct image name

Free up Disk Space Used by Docker

See this reference.

For issues related to the size of the Docker.qcow2 file, see this link.

Temporarily Install a Package for Troubleshooting

If you need to, you can probably simply install packages in any transient container, e.g.:

1
2
3
4
apt-get update
apt-get install netcat
nc -v -w3 10.10.10.100 5432
 

Run A Container With Env Variables for Testing

1
2
docker run -it -e  MY_ENV_VAR=somevalue myrepo/myapp:latest /bin/bash
 

Increase Terminal Width to View Truncated Lines

E.g. if output of ps is truncated at 80 characters by default. This happens because ps is using the width from the $COLUMNS variable. Specify a large value for $COLUMNS when using ps, e.g.:

1
2
root@ff41cab9438f:/# COLUMNS=1000 ps -aux
 

Remove Containers Created and Exited 2 Weeks ago

1
2
docker ps -a | grep "2 weeks" | grep "Exited" | awk '{print $1}' | sudo xargs docker rm
 

Remove All Exited Containers

1
2
docker ps -a | grep Exit | awk '{print $1}' |  xargs docker rm
 

Grep Docker stdout and stderr

E.g. if you have an application that’s writing to both streams and you’re not sure where logs are going:

1
2
docker logs 4017b8a68f98 2>&1 | grep
 

Keep a Container Running

E.g. if it starts and immediately exits:

1
2
sudo docker run -t -i my-registry/mycontainer /bin/bash
 

This will attach you directly to the container so you can check the logs.

1
2
sudo docker run -d my-registry/my-container /bin/bash
 

This will start it detached.

Run A Container With a Link to Another

1
2
docker run --rm -t -i --name mycontainer --link docker_consul_1 my-registry/mycontainer
 

Logs

Get the last 20 lines of logs:

1
2
docker logs --tail 20 fc805ce579d3
 

Other References

Cheat sheet.

View and edit this post on GitHub

Filed Under: quick reference Tagged With: docker

Java Collections Quick Reference

03/21/2015 by Luc

A cheat sheet for my frequently used/forgotten Java Collections snippets.

Quickly Create a List

1
2
3
4
import static java.util.Arrays.asList;
asList("hello", "goodbye");
import static java.util.Collections.unmodifiableList;
unmodifiableList(asList("hello", "goodbye"));

Empty Collection Syntax

1
List<String> requiredInputs = Collections.<String>emptyList();

Also see here.

Safe Empty Array

1
2
3
public static String[] safe(String[] other ) {
    return other == null ? new String[0] : other;
}

Convert Array to List

1
Arrays.asList(values)

Collection to Array

1
values.toArray(new String[values.size()])

Loop Over Keys And Values In A Map

1
2
3
4
5
6
for (Iterator<Map.Entry<String, Object>> it = resultsData.entrySet().iterator(); it.hasNext();) {
            Map.Entry<String, Object> entry = it.next();
            String key = entry.getKey();
            Object value = entry.getValue();
            System.out.println(String.format("key: %s, value: %s", key, value));
        }

Filed Under: code snippets, quick reference Tagged With: collections, java

Sed Quick Reference

09/17/2014 by Luc

A cheat sheet for my favourite sed commands.

[gist id = “9b185deb12d0e729689b”]

Filed Under: quick reference Tagged With: linux

Find Quick Reference

09/12/2014 by Luc

This is a quick cheat sheet of my favourite find commands.

Find Then Grep

find . -name file.* -exec grep -Hn thingtogrep {} \;

Find By Name

find . -name "MyCProgram.c"

find . -name "proto*"

Find And Copy Somewhere

find . -name <file> -exec cp {} <target dir>/ \;

find . -name wget* -exec cp {} $TEMP/wget \; (confirmed)

Find And Delete

find . -name ".svn" -exec rm -rf {} \;
This finds and deletes svn directories. See link on Cyberciti for more details.

Find And Open A File

find ~ -name "file.xml" -exec vim {} \;

Find And Move Somewhere

find -name "*.java" -exec mv {} /cygdrive/c/temp \;

Find And Replace In Files

find . -type f -print | xargs sed -i 's/thing_to_replace/another_thing/g'
Or
find . -type f -exec sed -i 's/ugly/beautiful/g' {} \;
Note the -i is for edit in place, not case-insensitive.

Run A Command On A File

find <CONDITION to Find files> -exec <OPERATION> \;

find -iname "MyCProgram.c" -exec rm {} \;

find -iname "MyCProgram.c" -exec md5sum {} \;

Finding the Top 5 Big Files

find . -type f -exec ls -s {} \; | sort -n -r | head -5

find . -type f -exec ls -s {} \; | sort -n -r | head -10 > ~/largefiles.txt &

Find Files Based on File-Type

find . -type s

Show files which are modified after the specified file

find -newer ordinary_file

Find Files Larger Than The Given Size

find ~ -size +100M

Find Files Whose Content Was Updated Within The Last…

1 hour: find . -mmin -60

1 day: find / -mtime -1

Exclude a Directory

find .  -path './excludedir' -prune -o -name "*" -print

Search All jar Files In A Directory For A Class Name

find . -type f -name '*.jar' -print0 | xargs -n1 -0i sh -c 'jar tf "{}" | grep -q AQOracleDriver && echo "{}"'

Avoiding Permission denied messages

Add this:
-perm -a+r -perm /a+w ! -perm /a+x

Filed Under: quick reference Tagged With: linux

  • Page 1
  • Page 2
  • Next Page »

Copyright ©2019 Luc Russell | Site by Sproutee