Luc Russell

Data Engineering and Full Stack Development

Consuming from nsq to Hadoop HDFS with Apache Flume

by

I was working on a prototype for consuming from nsq to the Hadoop HDFS file system with Apache Flume. I couldn’t find much information, so I made a small project.

Below are a few extra notes about nsq and the project.

  • docker-flume/Dockerfile builds a container which includes Flume, nsq and Hadoop core necessary for persisting files to HDFS
  • In order to be able to quickly modify some files in the built docker container, some extra paths are mapped in the docker-compose.yml:
    • Adding the start-flume.sh script allows updating the classpath to quickly add libraries and restart the container.
    • Adding the flume/flume.conf path means the flume config can be adjusted and the container restarted, without requiring a container rebuild.
  • In flume.conf docker.sources.r1.command specifies nsq_tail. While this is a quick way to consume from nsq, it is possibly not a reliable technique to use in a production environment.
  • nsq doesn’t appear to have a mechanism for resuming from an offset, like Kafka. This means there could be gaps in data if a consumer goes down and comes back up. In this sense, nsq has similarities to high volume messaging systems like RabbitMQ or ZeroMQ, which provide fewer message durability guarantees than Kafka.

Pyenv Quick Reference

by

Contents

Pyenv is a tool for easily switching between multiple versions of Python. One way to use the tool is in conjunction with the pyenv-virtualenv and pyenv-virtualenvwrapper plugins.

Detailed instructions are available here and here. The following reference is a brief summary of some frequently used commands.

Install Pyenv, pyenv-virtualenv and pyenv-virtualenvwrapper

Pyenv Installer

First use the Pyenv Installer. The ‘Github Way’ of installing is easiest.

Edit your profile (i.e. ~/.bash_profile, ~/.zshenv, etc) and add the following, adjusting paths as necessary:

Then install the following two plugins:
– pyenv-virtualenv
– pyenv-virtualenvwrapper

Edit your profile again, adding the following:

Usage

List Available Python Versions

Install a specific version

Create A New virtualenv Based on a Specific Python Version

See the full guide here.

Customize pip.conf

You can set a project or virtualenv specific pip.conf by creating one in
$VIRTUAL_ENV/pip.conf when the virtualenv is active:

Add this:

Other Useful Commands

Command Description
lsvirtualenv List all the virtualenvs
echo $WORKON_HOME Show the directory where virtualenvs are located
workon myvirtualenv Set the current virtualenv to myvirtualenv
pyenv global Show the global Python version currently in use
pyenv versions Show which Python versions are installed
pyenv global 3.6.4 Switch to the 3.6.4 version if installed

Kafkacat Quick Reference

by

Contents

kafkacat is a useful tool for working with Kafka brokers. It’s quick and very simple to use. This is a cheatsheet of some useful commands.

Produce

Consume

List Topics

Read Messages From File

Read messages from a file, with each message separated by a newline:

Generate a Test File of Messages

Given a file containing a DATE variable, generate a set of messages with incrementing timestamp, for use with the previous example:

Read the Last n Messages from Partition n

This example uses the -o flag to read from a specified offset. Specifying -3 allows retrieval of the last 3 messages. The -p option targets partition 0:

How to Set Up Kafka on Hortonworks Sandbox (Docker version)

by

Assumes a container was previously set up following these instructions and you now want to enable the Kafka service.

  1. Stop and delete existing an existing sandbox container if one was already created:

  2. Add the default Hortonworks Kafka port to the create_container.sh script:

  3. Recreate and start the container:

  4. SSH to the new container and start services. The password for the newly created container will have been reset to the hortonworks default (hadoop), so you’ll need to reset it at this point:

  5. When started, go to http://localhost:8080/#/login login as maria_dev/ maria_dev.

  6. Start the Kafka service from the menu.

  7. Check it works (assumes a local Kafka installation on your development machine).

    • In one terminal session:

    • In a second session:

Ethereum Dev Quick Reference

by

Set Up A Local Development Environment

Install and set up a local development account like this (Ubuntu):

Configure Local Environment to Create Test Ether

The --mine option produces test ether:

In another session, check logs for mining activity:

In dev mode, the node only mines if there are transactions. So to see activity here, send a transaction as below. Logging should then show:

Check How Many Accounts You Have

Go to the geth console and try a few commands:

The first account is referred to as the coinbase and can be accessed with eth.coinbase. This address will earn the mining rewards:

Check Balances and Transfer Between Accounts

This shows the commands for sending between two accounts and checking balances. Note you first need to unlock the account:

Install Solidity And Create a Simple Contract

Note that some Ethereum documentation is out of date, and refers to using Solidity directly from geth, which is not currently possible. See this issue. If you attempt this, you may see:

Install

There are several options for creating a contract on your local machine. One is to install the Solidity compiler, and import compiled files into geth. For Ubuntu:

Further instructions here:

Create A Contract

These instructions are summarized from here and here.

Create a new file, e.g. Test.sol:

Use a tool like this Line Break Removal Tool to remove newlines.

Compile (in a terminal session, not geth console):

Check what’s in test.js:

Load the file in the geth console:

Check the content of testOutput:

Create a contract with the content of testOutput:

Save the address if you want to later be able to run this contract from outside your geth session.

You should now be able to call the double function from within the geth session:

Use web3.py To Interact with Geth from Python

Assuming geth is running on localhost:8545, use web3.py to get some basic information from outside the geth console.

Note that in order to access some of the API methods, you need to specify an additional parameter to geth on startup: --rpcapi="db,eth,net,web3,personal,web3".

Without this, you may see an error like method personal_listAccounts does not exist/is not available.

Connect to a Local Development Environment With Remix