quick reference

Kafka Quick Reference

11/07/2016 by Luc

Create A Topic
Delete a Topic
Add a Partition
Describe a Topic
Reassign partitions
Checking Lag
How to Set Up librdkafka with Pykafka
Kafkacat

Create A Topic

./kafka-topics.sh --create --zookeeper myzookeeper:2181 --replication-factor 3 --partitions 3 --topic mytopic

1 2	./kafka-topics.sh --create --zookeeper myzookeeper:2181 --replication-factor 3 --partitions 3 --topic mytopic

Delete a Topic

./kafka-topics.sh --zookeeper myzookeeper:2181 --delete --topic mytopic

1 2	./kafka-topics.sh --zookeeper myzookeeper:2181 --delete --topic mytopic

Add a Partition

./kafka-topics.sh --zookeeper myzookeeper:2181 --alter --topic mytopic --partitions 2

1 2	./kafka-topics.sh --zookeeper myzookeeper:2181 --alter --topic mytopic --partitions 2

Describe a Topic

Note this describes all the topics, regardless of what topic is supplied.

./kafka-topics.sh --describe mytopic

1 2	./kafka-topics.sh --describe mytopic

Reassign Partitions

Create a file reassign.json:

{ "version":1, "partitions":
[{"topic":"my-topic", "partition":0, "replicas":[0,1,2] }] }

{ "version":1, "partitions":

[{"topic":"my-topic", "partition":0, "replicas":[0,1,2] }] }

Then execute:

./kafka-reassign-partitions.sh --zookeeper myzookeeper:2181 --reassignment-json-file reassign.json --execute

1 2	./kafka-reassign-partitions.sh --zookeeper myzookeeper:2181 --reassignment-json-file reassign.json --execute

Checking Lag

A simple solution for quickly checking lag is as follows. However, a longer term solution is to set up a Grafana instance and report lag to this using statsd.

./kafka-consumer-offset-checker.sh --zookeeper myzookeeper:2181 --group my_group_id --topic mytopic

1 2	./kafka-consumer-offset-checker.sh --zookeeper myzookeeper:2181 --group my_group_id --topic mytopic

How to Set Up librdkafka with Pykafka

Follow this reference.

In summary:

    git clone https://github.com/edenhill/librdkafka.git
    cd librdkafka/
    ./configure
    make
    sudo make install

git clone https://github.com/edenhill/librdkafka.git

cd librdkafka/

./configure

make

sudo make install

Then:

git clone https://github.com/Parsely/pykafka.git
cd pykafka
python setup.py develop

git clone https://github.com/Parsely/pykafka.git

cd pykafka

python setup.py develop

The last part, calling setup.py develop to put pykafka in develop mode, is necessary for creating the link between the two.

Kafkacat

kafkacat is a useful tool, here is a kafkacat quick reference.

Docker Quick Reference

10/22/2016 by Luc

Docker Compose Force Rebuild
Working With Docker for Mac
Get Into a Docker Container without docker-enter
Get Into a Docker Image which Fails to Start
Free up Disk Space Used by Docker
Temporarily Install a Package for Troubleshooting
Run A Container With Env Variables for Testing
Remove Containers Created and Exited 2 Weeks ago
Remove All Exited Containers
Grep Docker stdout and stderr
Keep a Container Running
Run A Container With a Link to Another
Logs
Other References

Docker Compose Force Rebuild

Also see here.

docker-compose rm -v -f
docker-compose build --no-cache
docker-compose up --force-recreate

docker-compose rm -v -f

docker-compose build --no-cache

docker-compose up --force-recreate

For compose files where you have trouble removing referenced images, try:

docker-compose down --rmi all

1 2	docker-compose down --rmi all

See Docker Compose reference.

Remove a Volume

docker volume ls
docker volume rm -f some_grafana_data

docker volume ls

docker volume rm -f some_grafana_data

You might then see:

Error response from daemon: unable to remove volume: remove some_grafana_data: volume is in use - [4633083c16d8cab09af2163a2e76105ce1a7e2a5e2d23289f7b67125c607c9a6]

1 2	Error response from daemon: unable to remove volume: remove some_grafana_data: volume is in use - [4633083c16d8cab09af2163a2e76105ce1a7e2a5e2d23289f7b67125c607c9a6]

In which case:

docker rm -f 4633083c16d8cab09af2163
docker volume rm -f some_grafana_data

docker rm -f 4633083c16d8cab09af2163

docker volume rm -f some_grafana_data

Working With Docker for Mac

Aliasing 0.0.0.0 to work around dynamic IPs

To solve this problem:

I want to connect from a container to a service on the host. The
mac has a changing IP address or no address if you have no network access

The current recommendation is to attach an unused IP to the lo0 interface on the Mac e.g.

sudo ifconfig lo0 alias 10.200.10.1/24

1 2	sudo ifconfig lo0 alias 10.200.10.1/24

Make sure your service is listening on this address or 0.0.0.0
(i.e. not 127.0.0.1). Then any containers which need to access the
service can use the 10.200.10.1 address.

Accessing The Xyve Virtual Machine

You might need to do this after restarting/upgrading docker. To get into the xyve vm use screen:

screen ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty

1 2	screen ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty

The login is ‘root’, no password.

Add an Internal Company Registry or Repo

vi /etc/hosts

1 2	vi /etc/hosts

and add the address e.g.:

10.10.10.100   my-registry.mycompany.com

1 2	10.10.10.100 my-registry.mycompany.com

Exit screen with Ctrl-a then k.

Here is a screen reference: http://ss64.com/osx/screen.html.

Get Into a Docker Container without docker-enter

docker exec -it 963ffd75c1b5 /bin/bash

1 2	docker exec -it 963ffd75c1b5 /bin/bash

Get Into a Docker Image which Fails to Start

docker run -it applications_mycontainer /bin/bash

1 2	docker run -it applications_mycontainer /bin/bash

Use docker images to find the correct image name

Free up Disk Space Used by Docker

See this reference.

For issues related to the size of the Docker.qcow2 file, see this link.

Temporarily Install a Package for Troubleshooting

If you need to, you can probably simply install packages in any transient container, e.g.:

apt-get update
apt-get install netcat
nc -v -w3 10.10.10.100 5432

apt-get update

apt-get install netcat

nc -v -w3 10.10.10.100 5432

Run A Container With Env Variables for Testing

docker run -it -e  MY_ENV_VAR=somevalue myrepo/myapp:latest /bin/bash

1 2	docker run -it -e MY_ENV_VAR=somevalue myrepo/myapp:latest /bin/bash

Increase Terminal Width to View Truncated Lines

E.g. if output of ps is truncated at 80 characters by default. This happens because ps is using the width from the $COLUMNS variable. Specify a large value for $COLUMNS when using ps, e.g.:

root@ff41cab9438f:/# COLUMNS=1000 ps -aux

1 2	root@ff41cab9438f:/# COLUMNS=1000 ps -aux

Remove Containers Created and Exited 2 Weeks ago

docker ps -a | grep "2 weeks" | grep "Exited" | awk '{print $1}' | sudo xargs docker rm

1 2	docker ps -a \| grep "2 weeks" \| grep "Exited" \| awk '{print $1}' \| sudo xargs docker rm

Remove All Exited Containers

docker ps -a | grep Exit | awk '{print $1}' |  xargs docker rm

1 2	docker ps -a \| grep Exit \| awk '{print $1}' \| xargs docker rm

Grep Docker stdout and stderr

E.g. if you have an application that’s writing to both streams and you’re not sure where logs are going:

docker logs 4017b8a68f98 2>&1 | grep

1 2	docker logs 4017b8a68f98 2>&1 \| grep

Keep a Container Running

E.g. if it starts and immediately exits:

sudo docker run -t -i my-registry/mycontainer /bin/bash

1 2	sudo docker run -t -i my-registry/mycontainer /bin/bash

This will attach you directly to the container so you can check the logs.

sudo docker run -d my-registry/my-container /bin/bash

1 2	sudo docker run -d my-registry/my-container /bin/bash

This will start it detached.

Run A Container With a Link to Another

docker run --rm -t -i --name mycontainer --link docker_consul_1 my-registry/mycontainer

1 2	docker run --rm -t -i --name mycontainer --link docker_consul_1 my-registry/mycontainer

Logs

Get the last 20 lines of logs:

docker logs --tail 20 fc805ce579d3

1 2	docker logs --tail 20 fc805ce579d3

Other References

Cheat sheet.

View and edit this post on GitHub

Java Collections Quick Reference

03/21/2015 by Luc

A cheat sheet for my frequently used/forgotten Java Collections snippets.

Quickly Create a List

import static java.util.Arrays.asList;
asList("hello", "goodbye");
import static java.util.Collections.unmodifiableList;
unmodifiableList(asList("hello", "goodbye"));

import static java.util.Arrays.asList;

asList("hello", "goodbye");

import static java.util.Collections.unmodifiableList;

unmodifiableList(asList("hello", "goodbye"));

Empty Collection Syntax

List<String> requiredInputs = Collections.<String>emptyList();

1	List<String> requiredInputs = Collections.<String>emptyList();

Also see here.

Safe Empty Array

public static String[] safe(String[] other ) {
    return other == null ? new String[0] : other;
}

public static String[] safe(String[] other ) {

return other == null ? new String[0] : other;

}

Convert Array to List

Arrays.asList(values)

1	Arrays.asList(values)

Collection to Array

values.toArray(new String[values.size()])

1	values.toArray(new String[values.size()])

Loop Over Keys And Values In A Map

for (Iterator<Map.Entry<String, Object>> it = resultsData.entrySet().iterator(); it.hasNext();) {
            Map.Entry<String, Object> entry = it.next();
            String key = entry.getKey();
            Object value = entry.getValue();
            System.out.println(String.format("key: %s, value: %s", key, value));
        }

for (Iterator<Map.Entry<String, Object>> it = resultsData.entrySet().iterator(); it.hasNext();) {

Map.Entry<String, Object> entry = it.next();

String key = entry.getKey();

Object value = entry.getValue();

System.out.println(String.format("key: %s, value: %s", key, value));

}

Sed Quick Reference

09/17/2014 by Luc

A cheat sheet for my favourite sed commands.

[gist id = “9b185deb12d0e729689b”]

Find Quick Reference

09/12/2014 by Luc

This is a quick cheat sheet of my favourite find commands.

Find Then Grep

find . -name file.* -exec grep -Hn thingtogrep {} \;

Find By Name

find . -name "MyCProgram.c"

find . -name "proto*"

Find And Copy Somewhere

find . -name <file> -exec cp {} <target dir>/ \;

find . -name wget* -exec cp {} $TEMP/wget \; (confirmed)

Find And Delete

find . -name ".svn" -exec rm -rf {} \;
This finds and deletes svn directories. See link on Cyberciti for more details.

Find And Open A File

find ~ -name "file.xml" -exec vim {} \;

Find And Move Somewhere

find -name "*.java" -exec mv {} /cygdrive/c/temp \;

Find And Replace In Files

find . -type f -print | xargs sed -i 's/thing_to_replace/another_thing/g'
Or
find . -type f -exec sed -i 's/ugly/beautiful/g' {} \;
Note the -i is for edit in place, not case-insensitive.

Run A Command On A File

find <CONDITION to Find files> -exec <OPERATION> \;

find -iname "MyCProgram.c" -exec rm {} \;

find -iname "MyCProgram.c" -exec md5sum {} \;

Finding the Top 5 Big Files

find . -type f -exec ls -s {} \; | sort -n -r | head -5

find . -type f -exec ls -s {} \; | sort -n -r | head -10 > ~/largefiles.txt &

Find Files Based on File-Type

find . -type s

Show files which are modified after the specified file

find -newer ordinary_file

Find Files Larger Than The Given Size

find ~ -size +100M

Find Files Whose Content Was Updated Within The Last…

1 hour: find . -mmin -60

1 day: find / -mtime -1

Exclude a Directory

find . -path './excludedir' -prune -o -name "*" -print

Search All jar Files In A Directory For A Class Name

find . -type f -name '*.jar' -print0 | xargs -n1 -0i sh -c 'jar tf "{}" | grep -q AQOracleDriver && echo "{}"'

Avoiding Permission denied messages

Add this:
-perm -a+r -perm /a+w ! -perm /a+x