[Note] Message Queue S/W in Docker

 

IBM MQ

$ docker volume create mqdata
$ docker run --name ibmmq --hostname ibmmq \
    -e LICENSE=accept -e MQ_QMGR_NAME=QM1 -e MQ_APP_PASSWORD=passw0rd \
    -v mqdata:/data/docker.data/mq -p 1414:1414 -p 9443:9443 -d ibmcom/mq:latest

Above will set up followings:

• Queue manager QM1
• Queue DEV.QUEUE.1
• Channel: DEV.APP.SVRCONN
• Listener: DEV.LISTENER.TCP on port 1414

Test 1:

$ docker exec -ti ibmmq bash
$ dspmqver      # version info
$ dspmq         # display running queue mgr

 

Test 2:

1. Open https://localhost:9443/ibmmq/console/ 
2. Log in with user ID and PW as “admin” and “passw0rd”.
3. Click on "Manage QM1" box
   
4. Click on DEV.QUEUE.1
5. Click on "Create" button - right nav will slide out. Type something in "Application Data" and click on "Create" button.  You'll see your message in the queue.
   
6. Click on trash can icon to clear up the queue.

References

• https://developer.ibm.com/components/ibm-mq/tutorials/mq-connect-app-queue-manager-containers
• https://developer.ibm.com/components/ibm-mq/articles/mq-downloads

RabbitMQ

$ docker run -d --hostname rabbitmq --name rabbitmq -e RABBITMQ_DEFAULT_USER=admin -e RABBITMQ_DEFAULT_PASS=passw0rd -p 15672:15672 -p 5672:5672 rabbitmq:3-management

Admin URL: http://localhost:15672

Redis

$ docker run --name redis --hostname redis -p 6379:6379 -d redis

Test:

$ docker exec -it redis bash
$ redis-cli
> ping
> set name pnap
> get name
> incr counter
> incr counter
> get counter
> exit

References

• https://hub.docker.com/_/redis/
  
• https://markheath.net/post/exploring-redis-with-docker
• https://phoenixnap.com/kb/docker-redis

 

 

[Note] Kafka in Docker

Needs

Simple Kafka in Docker set up for development.  Single broker in Kafka, and Zookeeper should be usable for other than Kafka.

Tried various methods following several blogs/tutorials but none of them worked or suitable for my needs.

 

Prerequisite

Docker is installed & running, docker-compose is installed.

My Environment

Ubuntu 20.04

Steps

1. Save above as docker-compose.yml
2. $ sudo docker-compose up --build -d
   

Test

1. Download Kafka to use Kafka commands
   https://kafka.apache.org/downloads 
2. Create topic
   $ ./kafka-topics.sh -zookeeper localhost:2181 --create --topic test --partitions 3 --replication-factor 1
3. List topics
   $ ./kafka-topics.sh --zookeeper localhost:2181 --list 
4. Describe topic
   $ ./kafka-topics.sh --zookeeper localhost:2181 --describe --topic test

Test Python code

$ pip install kafa-python

producer.py

from json import dumps

from kafka import KafkaProducer

producer = KafkaProducer(bootstrap_servers=['localhost:9092'],

value_serializer=lambda x:

dumps(x).encode('utf-8'))

for e in range(10):

data = {'number' : e}

producer.send('test', value=data)

 

consumer.py

from kafka import KafkaConsumer

from json import loads

consumer = KafkaConsumer(

'test',

bootstrap_servers=['localhost:9092'],

auto_offset_reset='earliest',

enable_auto_commit=True,

group_id='my-group',

value_deserializer=lambda x: loads(x.decode('utf-8')))

for message in consumer:

message = message.value

print(message)

Run publisher first, and then consumer.

REFERENCE

• Kafka/Zookeeper Docker Images: https://hub.docker.com/r/wurstmeister/kafka/
• Docker kafrka, code, https://github.com/wurstmeister/kafka-docker
• https://www.bennettnotes.com/post/setup-kafka-zookeeper-in-docker/
• https://analyticshut.com/manage-apache-kafka-topics/ 
  

READINGS

• Kafka, https://aws.amazon.com/msk/what-is-kafka/
• Zookepper, https://data-flair.training/blogs/zookeeper-tutorial/
• Docker-app, https://towardsdatascience.com/an-end-to-end-machine-learning-project-with-python-pandas-keras-flask-docker-and-heroku-c987018c42c7
• python-kafka
• https://linuxhint.com/read_data_kafka_python/
• https://towardsdatascience.com/kafka-python-explained-in-10-lines-of-code-800e3e07dad1

 

[Note] 802.11ac 2.4G/5G USB adapter for Ubuntu

Not many AC wifi USB adapters are for Linux.  This one is:

https://www.amazon.com/gp/product/B08BYBL2TQ/

 

Good price, $19.97 (as of today, 2020-Nov).  

Relatively easy to use.  Read some comments and found how to install it on Linux:

1. git clone this, https://github.com/RinCat/RTL88x2BU-Linux-Driver
2. make
3. sudo make install
4. sudo vi /etc/modules, and add "88x2bu" if not added already
5. sudo modprobe 88x2bu or reboot
   

 

 

[Note] Video capture adapter install on Ubuntu 20.04

[This is just a installation note.]

 

Video Capture

 

 

Product Page, https://www.amazon.com/gp/product/B01E5ITE2W/

It's only $11, USB2.0 video capture USB adapter.  It's to transfer old videos, not high resolution video capture purpose.

No driver is needed.  Use OBS Studio and it will be listed.  Some helpful comments from the product review:

Works great with Ubuntu Studio Linux!
By Kelly on October 10, 2020 I purchased this device to archive some old VHS tapes using an S-VHS machine. I plugged it into my linux box, and without any fanfare or problems, it just worked. VERY HAPPY! It shows up as Video to USB adapter in OBS. Smooth frame rate, and sharp image quality. If you have an S-VHS deck, use the S-Video input. It gets rid of all the dot crawl and soft edges. Would recommend for sure! 

Works great under linux for digitizing old analog video tapes
By Jacob D. on November 20, 2017 There appear to be be a ton of variations on this same product, but I opted for this particular one due to the great price and good reviews. I bought this to digitize a bunch of old family videos on Sony Hi8 tapes that were 30 years old and beginning to degrade and no longer play. I was hoping to get digital versions of these old tapes before they stopped playing completely, and this worked perfect for that.
My setup might be a little different than others since I'm running linux (kubuntu) as my operating system, so I didn't try any of the included drivers or software, but I'm happy to report that this functions perfectly as a capture device under my linux version with no additional configuration or driver compilation / installation. I used VLC to preview via the inputs and capture to a file, and after playing around with some different codecs, found some settings that provided a good balance between file size and quality.
Physically, it was simple to plug in and connect and disconnect everything between capture sessions, the color-coded cable extensions make it a breeze to connect the different wires. On my laptop, I had an easily accessible usb port so it wasn't a problem to plug this in and have it stick out the side. The capture device doesn't have too high of a resolution, which isn't really necessary for the older analog sources, so it still works great. I successfully tried both the s-video and yellow rca for video, and didn't have any problem with the red/white rca audio inputs.
To sum up, this worked perfectly for my needs under linux, and it's an amazing value for the price!

Simple, cheap, and "good enough" for some capture projects
By Jonathan Campbell on November 11, 2019 Pros:
- Works flawlessly with Linux. Appears as UVC class device and ALSA audio capture device.
- Audio and video capture works, comes in as 720x480 UYVY uncompressed at full frame rate and should work with any video capture software using Video4Linux2.
Cons:
- Appears to be deinterlacing the video where there is motion in the frame, meaning fields are averaged together when things move. If you're expecting to capture the full interlaced video for full quality this might not be the ideal capture device for you. If that's not a concern for you, go for it.
- As stated elsewhere, loses track of color subcarrier when given video with Macrovision.
- If you start capture with no signal, then turn on the VCR and start playing, the captured video will be very bright until you close the capture program and reopen it again. Workaround is to turn on the VCR and give it a signal, then open the capture program and start capturing.

 

 

 

 

 

Playing with FPGA (simulator)

Although I've wanted to try out FPGA for a long time, but never really had the chance/time to.  This is a note on learning/playing with FPGA.

In this posting, which is just the first step -- learning about it, and use simulator first.  If/when I decide to continue with this journey, I'll then buy a small FPGA board, and continue.

I've gathered the information by talking to some experts online, googling, and read a few books.

What is FPGA (Field Programmable Gate Array)

First, what FPGA is.  Watch this short video - only 1:25 long -

Or, read Wiki, https://en.wikipedia.org/wiki/Field-programmable_gate_array

Installing Simulator

There are several simulators, and someone suggested to use Icarus Verilog (http://iverilog.icarus.com/) as it is light. 

Environment: Ubuntu 20.04

I've originally downloaded the source code and compiled, but later found Ubuntu already has apt package ready.  

Reference video, https://www.youtube.com/watch?v=xd-ZvBJiv3M -- this is in Chinese, so no clue what the video uploader is saying, but you can just watch the video with steps.  I wrote the steps here to avoid watching the whole thing.

$ sudo apt-get install iverilog

$ sudo apt-get install gtkwave

 

Testing Simulator #1

1. Create project directory

$ mkdir verilog_test1

2. Using VSC, edit a file called "test1.v" in the directory created above:

module main;

initial

begin

$display("Hello world!");

$finish;

end

endmodule

3. compile the code

$ iverilog test1.v -o test1

4. Run it

$ vvp test1

 

Testing Simulator #2

This is from https://www.youtube.com/watch?v=3Xm6fgKAO94, and it also shows installing Icarus Verilog on Windows.

1. Create two files

hello.v:

// hello.v

module hello(A, B);

input A;

output B;

assign B = A;

endmodule

hello_tb.v:

// hello_tb.v

`timescale 1ns / 1ns

`include "hello.v"

module hello_tb;

reg A;

wire B;

hello uut(A, B);

initial begin

$dumpfile("hello_tb.vcd");

$dumpvars(0, hello_tb);

A = 0; // single bit

#20; // wait for 20ns, to view wave form

A = 1;

#20;

A = 0;

#20;

$display("Test complete");

end

endmodule

2. Compile and run

$ iverilog -o hello_tb.vvp hello_tb.v
$ vvp hello_tb.vvp
VCD info: dumpfile hello_tb.vcd opened for output.
Test complete

hello_tb.vvp file is created.

 

3. View wave

$  gtkwave hello_tb.vcd

or, File > Open Tab > and select "hello_tb.vcd"

In left nav, lower pane, select both reg A, wire B.  Then click on "Insert" button.

 

Learning Resources

• https://www.fpga4fun.com/FPGAinfo1.html