Luc Russell
Data Engineering and Full Stack Development
  • Home
  • Blog

Luc / how to's / kafka, nsq /

Consuming from nsq to Hadoop HDFS with Apache Flume

10/25/2018

I was working on a prototype for consuming from nsq to the Hadoop HDFS file system with Apache Flume. I couldn’t find much information, so I made a small project. Below are a few extra notes about nsq and the project. docker-flume/Dockerfile builds a container which includes Flume, nsq and Hadoop core necessary for persisting […]

Luc / how to's / python /

Pyenv Quick Reference

12/04/2017

Contents Install Pyenv, pyenv-virtualenv and pyenv-virtualenvwrapper Pyenv Installer Usage List Available Python Versions Install a specific version Create A New virtualenv Based on a Specific Python Version Customize pip.conf Other Useful Commands Pyenv is a tool for easily switching between multiple versions of Python. One way to use the tool is in conjunction with the […]

Luc / kafka / kafka /

Kafkacat Quick Reference

08/23/2017

Contents Produce Consume List Topics Read Messages From File Generate a Test File of Messages Read the Last n Messages from Partition n kafkacat is a useful tool for working with Kafka brokers. It’s quick and very simple to use. This is a cheatsheet of some useful commands. Produce $ echo “foo” | kafkacat -b […]

Luc / kafka / kafka /

How to Set Up Kafka on Hortonworks Sandbox (Docker version)

03/24/2017

Assumes a container was previously set up following these instructions and you now want to enable the Kafka service. Stop and delete existing an existing sandbox container if one was already created: docker stop sandbox docker rm sandbox Add the default Hortonworks Kafka port to the create_container.sh script: -p 6667:6667 Recreate and start the container: […]

Luc / postgres / postgres /

PostgreSQL Quick Reference

01/01/2017

Connect $ psql -h localhost -d mydb -U myuser List Databases From within a psql session: l List Tables dt Connect To DB c myuser Execute SQL in a File $ psql -U myuser -d mydb -a -f myfile.sql Run a Query $ psql -U myuser -d mydb -c ‘SELECT * FROM my_table’ Output to […]

Luc / errors and problems / Errors and problems, kafka /

Kafka Troubleshooting Notes

11/07/2016

Something Wrong With One Broker in a Cluster Problem Consumer connected to a broker but not receiving messages. Looking at various topics, broker 0 is marked as an out of sync replica, e.g.: ./kafka-topics.sh –describe my-topic –zookeeper myzk:2181/kafka Topic: my-topic Partition: 6 Leader: 4 Replicas: 0,4,1 Isr: 4,1 … Topic: __consumer_offsets Partition: 1 Leader: 2 […]

Luc / quick reference / kafka /

Kafka Quick Reference

11/07/2016

Create A Topic Delete a Topic Add a Partition Describe a Topic Reassign partitions Checking Lag How to Set Up librdkafka with Pykafka Kafkacat Create A Topic ./kafka-topics.sh –create –zookeeper myzookeeper:2181 –replication-factor 3 –partitions 3 –topic mytopic Delete a Topic ./kafka-topics.sh –zookeeper myzookeeper:2181 –delete –topic mytopic Add a Partition ./kafka-topics.sh –zookeeper myzookeeper:2181 –alter –topic mytopic […]

Luc / quick reference / docker /

Docker Quick Reference

10/22/2016

Table of Contents Docker Compose Force Rebuild Working With Docker for Mac Aliasing 0.0.0.0 to work around dynamic IPs Accessing The Xyve Virtual Machine Add an Internal Company Registry or Repo Get Into a Docker Container without docker-enter Get Into a Docker Image which Fails to Start Free up Disk Space Used by Docker Temporarily […]

Luc / errors and problems / postgres /

sqlalchemy.exc.OperationalError

01/01/2016

Symptom I encountered this error while trying to run commands on Postgres running in a Docker container: sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) FATAL: could not write init file Solution Ensure Docker has enough disk space, see Docker Quick Reference.

Luc / errors and problems / python /

SQLAlchemy “is not” behaviour

10/26/2015

If you want to use != but lint complains, use isnot instead. See this reference for more details.

1 2 3 4 ›»

Back to Top

© Luc Russell 2020 · Site by Sproutee