I was working on a prototype for consuming from nsq to the Hadoop HDFS file system with Apache Flume. I couldn’t find much information, so I made a small project. Below are a few extra notes about nsq and the project. docker-flume/Dockerfile builds a container which includes Flume, nsq and Hadoop core necessary for persisting […]
Contents Install Pyenv, pyenv-virtualenv and pyenv-virtualenvwrapper Pyenv Installer Usage List Available Python Versions Install a specific version Create A New virtualenv Based on a Specific Python Version Customize pip.conf Other Useful Commands Pyenv is a tool for easily switching between multiple versions of Python. One way to use the tool is in conjunction with the […]
Contents Produce Consume List Topics Read Messages From File Generate a Test File of Messages Read the Last n Messages from Partition n kafkacat is a useful tool for working with Kafka brokers. It’s quick and very simple to use. This is a cheatsheet of some useful commands. Produce $ echo “foo” | kafkacat -b […]
Assumes a container was previously set up following these instructions and you now want to enable the Kafka service. Stop and delete existing an existing sandbox container if one was already created: docker stop sandbox docker rm sandbox Add the default Hortonworks Kafka port to the create_container.sh script: -p 6667:6667 Recreate and start the container: […]
Connect $ psql -h localhost -d mydb -U myuser List Databases From within a psql session: l List Tables dt Connect To DB c myuser Execute SQL in a File $ psql -U myuser -d mydb -a -f myfile.sql Run a Query $ psql -U myuser -d mydb -c ‘SELECT * FROM my_table’ Output to […]
Luc / errors and problems / Errors and problems, kafka /
Something Wrong With One Broker in a Cluster Problem Consumer connected to a broker but not receiving messages. Looking at various topics, broker 0 is marked as an out of sync replica, e.g.: ./kafka-topics.sh –describe my-topic –zookeeper myzk:2181/kafka Topic: my-topic Partition: 6 Leader: 4 Replicas: 0,4,1 Isr: 4,1 … Topic: __consumer_offsets Partition: 1 Leader: 2 […]
Luc / quick reference / kafka /
Create A Topic Delete a Topic Add a Partition Describe a Topic Reassign partitions Checking Lag How to Set Up librdkafka with Pykafka Kafkacat Create A Topic ./kafka-topics.sh –create –zookeeper myzookeeper:2181 –replication-factor 3 –partitions 3 –topic mytopic Delete a Topic ./kafka-topics.sh –zookeeper myzookeeper:2181 –delete –topic mytopic Add a Partition ./kafka-topics.sh –zookeeper myzookeeper:2181 –alter –topic mytopic […]
Luc / quick reference / docker /
Table of Contents Docker Compose Force Rebuild Working With Docker for Mac Aliasing 0.0.0.0 to work around dynamic IPs Accessing The Xyve Virtual Machine Add an Internal Company Registry or Repo Get Into a Docker Container without docker-enter Get Into a Docker Image which Fails to Start Free up Disk Space Used by Docker Temporarily […]
Luc / errors and problems / postgres /
Symptom I encountered this error while trying to run commands on Postgres running in a Docker container: sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) FATAL: could not write init file Solution Ensure Docker has enough disk space, see Docker Quick Reference.
Luc / errors and problems / python /
If you want to use != but lint complains, use isnot instead. See this reference for more details.