December 15, 2019

[OwnServer] back up Google Drive, Google Photos, One Drive

This is a note on backing up Google Drive, Google Photos, One Drive.

UPDATE:
'rclone' alone can do all of below.  Since I tried different methods, I'm leaving it here for a reference purpose.  I now use rclone for all of the cloud storage sync, without any cloud specific clients.  And there is a software called 'rclone browser' for linux (and Windows) that lets you browse directories and files the cloud storage.



Usage of Cloud (online storage with sync feature):
  • Google Drive: I use paid Google drive, which also allows original file size to be backed up to Google drive/photos.  Family members share documents using Google docs.
  • Google Photos: My kids use google photos, free version for backing up for their photos on their devices.  
  • One Drive: Kids use One Drive to back up their PC files.

Environment: Ubuntu 16

One Drive

Use 'rclone'

Install

This is how I installed on my Ubuntu 16.

$ curl -O https://downloads.rclone.org/rclone-current-linux-amd64.zip
$ unzip rclone-current-linux-amd64.zip
$ cd rclone-*-linux-amd64
$ sudo cp rclone /usr/bin/
$ sudo chown root:root /usr/bin/rclone
$ sudo chmod 755 /usr/bin/rclone
$ sudo mkdir -p /usr/local/share/man/man1
$ sudo cp rclone.1 /usr/local/share/man/man1/
$ sudo mandb


Also, install the binary on Windows.  Since my linux box is headless, I need a web browser for auth thus using browser on Windows.

Set up

On linux side:
$ rclone config
> select 'n' for new remote
> type name, 'onedrive-keith'
> select '22' for one drive storage
> enter blank for client id, client secret
> advanced set up -> select 'n'
> auto config -> select 'n'
> will wait for "result" from windows


On windows side:
C:\> rclone authorize "onedrive"
> browser will pop up, give permission to onedrive with your account by clicking on "Yes"
> go back to cmd, and will give some result
> copy and paste the result into linux console


Back to linux:
> select '1' for onedrive personal
> select '0' for personal
> select 'y' for "Found drive 'root' of type 'personal', URL:...."
> select 'y' for "Yes this is ok"

Usage

e.g. added to cron entry similar to this:

$ rclone sync -P --create-empty-src-dirs onedrive-keith: /mnt/data1/OneDrive/keith

To install, just follow the "One Drive Doc" steps.  Once it's configured by running: rclone config, the configuration is stored at /home/kkim/.config/rclone/rclone.conf

Above "onedrive-keith" is just the configuration entry name.


Google Photos

I don't remember why I did not use rclone for Google Photos -- it's been a while, but I used another s/w called 'gphotos-sync'.

Read this doc, https://www.linuxuprising.com/2019/06/how-to-backup-google-photos-to-your.html

On Ubuntu 16, I had Python 3.5, and it requires 3.6+.  Below is removing 3.5 and installing 3.7:
$ sudo apt-get remove --auto-remove python3.5
--- install python3.6 or 3.7
https://www.osetc.com/en/how-to-install-the-latest-python-3-7-on-ubuntu-16-04-or-18-04.html
$ sudo apt update
$ sudo apt install software-properties-common
$ sudo add-apt-repository ppa:deadsnakes/ppa
$ sudo apt update
$ sudo apt install python3.7
$ python3.7 -v

Install pip3
$ curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
$ sudo python3.7 get-pip.py


And install gphotos-sync
$ sudo pip install gphotos-sync

Configuration is well explained in the mentioned web page (and it's several steps) -- it requires setting up Google API with each one's account.

Once configured in Google API, download client secret JSON file and stored on Ubuntu.  This will also require to use Windows (or any desktop with GUI) browser to configure Google API and for auth.

Usage

$ gphoto-sync [backup path] --secret [client_secret.json]


Google Drive

Again, I don't remember why I decided to use another software and not rclone.  I used 'grive'.  Similar to Google Drive, One Drive, it requires using web browser to configure API and get authenticated.

See this page for installation and configuration steps:
https://www.howtoforge.com/tutorial/sync-documents-with-google-drive-on-ubuntu-linux/

Source code is at:
https://github.com/prasmussen/gdrive

Install Steps

$ sudo apt-add-repository ppa:nilarimogard/webupd8
$ sudo apt-get update
$ sudo apt-get install grive


Usage

After set-up is done, get client ID and client secret strings. 

$ pushd [BACKUP PATH]
$ grive -i [CLIENT ID] -e [CLIENT SECRET] sync
$ popd



I've created bash scripts to simplify above and added to cron.  So the backup of backup is available on my home server.  If there is any issue with internet, or any of the cloud services, devices are lost or corrupted, the files are available on my server.

About Backup

From my experience, having (at least) 3 copies is the safest practice.  One on the device being used, and two additional backups:

10's-20's -- on original device, cloud, backup server, external HDD, USB flash drive
00's-10's --  on original device, cloud, external HDD, USB flash drive
90's-00's -- on original device, external HDD, CD/DVD
80's-90's -- on original device, floppy, tape drive

I tried complete online backup solution, and I think it's not worth it.  The software degrades the computer performance, back up is slow (over the net), and vendor lock-in is an issue.  Store on cloud just some files, and using external HDD/local server for complete backup is most practical solution.


[NOTE] Rust + Eclipse, Learning

Install IDE

Installing Eclipse for Rust, not as extension to existing Eclipse installation.

Download from, https://www.eclipse.org/downloads/packages/release/2019-09/r/eclipse-ide-rust-developers-includes-incubating-components

As long as the Rust env vars are set correction, nothing to configure.

Learning

https://doc.rust-lang.org/book/title-page.html

Important Facts

When I evaluate new (to me) languages, I first check for followings:
  • Script, VM, or native
  • Thread supporting
  • Easy of Syntax
  • OOP, Functional
  • GUI support
  • Libraries: web, network, AI
  • Performance
  • Benefits over other comparable languages
  • Support and popularity
Rust generates native binary.  Comparable languages: C++, Go lang.  It seems faster than g++ and go lang.  It is quite popular, although I didn't hear anyone I know or heard using it.  It's also multi-paradigm - can do OOP and functional.  

So far, I got pretty good impression with Rust from just spending a few days over the weekend with Rust.  Simple/easy to install, and start coding, a lot of learning materials, good amount of sample code, good open source projects.  Compiling and running are very easy.

I just started to learn Rust -- so I'll see how it goes.

Resources


December 14, 2019

[NOTE] Rust install on Windows in non-default path

Default paths for Rust are:
C:\Users\kkim\.cargo\bin
C:\Users\kkim\.rustup

To change this, just set up environment variables.
  1. Open https://www.rust-lang.org/tools/install
  2. Download rustup-init.exe
  3. Open cmd in the same directory where the download executable is.
  4. Set up environment variables as shown below and run the installer. 
    Remember to set these two env vars in system properties, not just within CMD.  These vars are needed as it's non-default locations.
  kkim@CHERRY E:\Downloads\2019-12-01
> set CARGO_HOME=c:\opt\rust\cargo

kkim@CHERRY E:\Downloads\2019-12-01
> set RUSTUP_HOME=c:\opt\rust\rustup

kkim@CHERRY E:\Downloads\2019-12-01
> rustup-init.exe

Welcome to Rust!

This will download and install the official compiler for the Rust
programming language, and its package manager, Cargo.

It will add the cargo, rustc, rustup and other commands to
Cargo's bin directory, located at:

  c:\opt\rust\cargo\bin

This can be modified with the CARGO_HOME environment variable.

Rustup metadata and toolchains will be installed into the Rustup
home directory, located at:

  c:\opt\rust\rustup

This can be modified with the RUSTUP_HOME environment variable.

This path will then be added to your PATH environment variable by
modifying the HKEY_CURRENT_USER/Environment/PATH registry key.

You can uninstall at any time with rustup self uninstall and
these changes will be reverted.

Current installation options:


   default host triple: x86_64-pc-windows-msvc
     default toolchain: stable
               profile: default
  modify PATH variable: yes

1) Proceed with installation (default)
2) Customize installation
3) Cancel installation
>2

I'm going to ask you the value of each of these installation options.
You may simply press the Enter key to leave unchanged.

Default host triple?


Default toolchain? (stable/beta/nightly/none)


Profile (which tools and data to install)? (minimal/default/complete)
complete

Modify PATH variable? (y/n)
y


Current installation options:


   default host triple: x86_64-pc-windows-msvc
     default toolchain: stable
               profile: complete
  modify PATH variable: yes


1) Proceed with installation (default)
2) Customize installation
3) Cancel installation
>1

info: profile set to 'complete'
info: syncing channel updates for 'stable-x86_64-pc-windows-msvc'
info: latest update on 2019-11-07, rust version 1.39.0 (4560ea788 2019-11-04)
warning: Force-skipping unavailable component 'lldb-preview-x86_64-pc-windows-msvc'
warning: Force-skipping unavailable component 'miri-x86_64-pc-windows-msvc'
info: downloading component 'cargo'
info: downloading component 'clippy'
info: downloading component 'llvm-tools-preview'
info: downloading component 'rls'
info: downloading component 'rust-analysis'
info: downloading component 'rust-docs'
 11.8 MiB /  11.8 MiB (100 %)  11.4 MiB/s in  1s ETA:  0s
info: downloading component 'rust-src'
info: downloading component 'rust-std'
180.7 MiB / 180.7 MiB (100 %)  12.0 MiB/s in 15s ETA:  0s
info: downloading component 'rustc'
 41.4 MiB /  41.4 MiB (100 %)  12.1 MiB/s in  3s ETA:  0s
info: downloading component 'rustfmt'
info: installing component 'cargo'
info: installing component 'clippy'
info: installing component 'llvm-tools-preview'
 10.5 MiB /  10.5 MiB (100 %)   7.3 MiB/s in  1s ETA:  0s
info: installing component 'rls'
info: installing component 'rust-analysis'
info: installing component 'rust-docs'
 11.8 MiB /  11.8 MiB (100 %) 649.6 KiB/s in  1m 13s ETA:  0s
info: installing component 'rust-src'
  2.5 MiB /   2.5 MiB (100 %) 781.3 KiB/s in  3s ETA:  0s
info: installing component 'rust-std'
180.7 MiB / 180.7 MiB (100 %)  16.8 MiB/s in 14s ETA:  0s
info: installing component 'rustc'
 41.4 MiB /  41.4 MiB (100 %)   8.6 MiB/s in  5s ETA:  0s
info: installing component 'rustfmt'
info: default toolchain set to 'stable'

  stable installed - rustc 1.39.0 (4560ea788 2019-11-04)


Rust is installed now. Great!

To get started you need Cargo's bin directory (c:\opt\rust\cargo\bin) in your
PATH
environment variable. Future applications will automatically have the
correct environment, but you may need to restart your current shell.

Press the Enter key to continue.

November 27, 2019

[NOTE] Win10 - get the full message for "Unauthorized changes blocked"

Event Viewer > Application and Services > Microsoft > Windows > Windows Defender > Operational

Get the full message using PowerShell:

> Get-WinEvent -FilterHashtable @{logname=’Microsoft-Windows-Windows Defender/Operational’; id=1123} | Select-Object Message

I added "C:\ubin\" to PATH, and added all custom BAT files there.  Create 'blockedmessage.bat' as following and add to C:\ubin\ :

@echo off
REM *** KK-20191203
REM *** list full blocked message by defender
powershell -command "Get-WinEvent -FilterHashtable @{logname='Microsoft-Windows-Windows Defender/Operational'; id=1123} | Select-Object Message"

November 9, 2019

[Quantum Computer] Getting Started: QISKit SDK

Qiskit is developed by IBM.
https://qiskit.org/

This note is to show how to install Qiskit on Win10, run the simulator locally.

Installation note
https://developer.ibm.com/code/2017/05/17/developers-guide-to-quantum-qiskit-sdk/
https://qiskit.org/documentation/install.html

Requirements:

  • Windows
  • Anaconda

Create environment

> conda create --name qiskit python=3.7
> activate qiskit

Install Qiskit

> pip install qiskit
> pip install qiskit-terra[visualization]



Unlike QDK, Qiskit is Python, which is a big plus -- no need to learn new language.

November 8, 2019

[Quantum Computer] Getting Started: QDK

This is just a note on installing QDK, Python, Jupyter on Windows.

QDK is Microsoft Quantum Development Kit.
https://www.microsoft.com/en-us/quantum/development-kit

Installation steps are here, https://docs.microsoft.com/en-us/quantum/install-guide/

But I had some minor trouble installing it, so this is the note.

Requirements:
  • Win10
  • Anaconda

[1] Install .NET package, 2.2.0


First, check if you have DOT NET package installed.  Do where dotnet and see if dotnet command is available.

If not, go to this site and install 2.2.  It's Important that current QDK requires 2.2.x version.
https://dotnet.microsoft.com/download/dotnet-core

Check if you have the right version:

> dotnet --list-sdks
2.2.402 [C:\Program Files\dotnet\sdk]
3.0.100 [C:\Program Files\dotnet\sdk]

As shown above, I have 2.2.402 installed.

Now install .NET tools:

> dotnet tool install -g Microsoft.Quantum.IQSharp
> dotnet iqsharp install

[2] Anaconda + Python + Jupyter

Run Anaconda and create environment, 'qsharp':

> conda create --name qsharp
> conda activate qsharp

Create workspace directory

> cd /d e:\dev\code\
> mkdir qsharp
> cd qhsharp

And install jupyter and qsharp

> conda install jupyter
> pip install qsharp
> conda install ipykernel

Check if qsharp kernel is installed:

> jupyter kernelspec list
Available kernels:
  python3    C:\opt\Anaconda3\envs\qsharp\share\jupyter\kernels\python3
  iqsharp    C:\ProgramData\jupyter\kernels\iqsharp


Run jupyter notebook:

> jupyter notebook


This is just to show you installing QDK with right .NET version with Jupyter.  Jupyter can be utilize the notebook, but I don't see much benefit of using Jupyter in this case.  VS or VS code would be better choice as you still have to write the code in Q#.

There are a lot of docs and examples of Q# -- this is a big plus for learning.
Next time, I'll put some notes on VS, VS code and Q#.


November 3, 2019

[Quantum Computer] Getting Started: Microsoft LIQUi|>

There are several quantum computer simulators and programming environments.  I'm experimenting/learning with a few -- and this is a note on installation and learning resources.

For Microsoft LIQUi|>, I actually experiment with it a couple of years ago, back in 2017.  I'm going through my note and putting some information on this posting, before I put some note on other quantum computer simulation and programming tools I recently toying around with.

First read these pages,

[1] https://blogs.msdn.microsoft.com/uk_faculty_connection/2017/10/11/getting-started-with-liqui-and-quantum-computing/

[2] Microsoft LIQUi|> (Liquid), https://www.microsoft.com/en-us/research/project/language-integrated-quantum-operations-liqui/


Installation

On Windows, Visual Studio (community version works fine), and uses LIQUi|> own language and F#.  It also support Ubuntu and OSX, but I only tried on Windows 10.

See the installation steps here, http://stationq.github.io/Liquid/getting-started/

Essentially, just two installation steps:

Step 1, install VS community version, https://visualstudio.microsoft.com/vs/community/
Step 2, download ZIP from this GitHub, extract and run, https://github.com/StationQ/Liquid

My impression on this was -- installation and running were simple, but hard to learn, and not much learning resources.  I don't recommend using this.  Next posting, I'll post about other more recent simulators and SDKs that seems easier to learn.

REFERENCES

October 5, 2019

[NOTE] RAM disk on Win10

Purpose

Use portion of computer memory (RAM) as disk.  The usage is up to you, but I wouldn't use this to store browser cache or system temp.

Environment

  • Win10 Pro 64 bit, build version 1803
  • Memory 64 GB 

Download

Install

[1] Install ImDisk first

This application will give very basic RAM disk feature only.  Whatever it's written in RAM disk will not be saved and will be lost after restarting the machine.

Note: bring the old Windows Control Panel and you'll find "ImDisk Virtual Disk Driver"



[2] If you have ransomware protection on, disable it temporarily to allow ImDisk toolkit to make shortcuts on Desktop:
  1. Run "Windows Defender Security Center" > Virus & thread protection
  2. Click on "Ransomware protection" > turn it off
[3] Install ImDisk toolkit
This will create icons on desktop, that's the reason for step #2.

[4] Turn back on ransomware protection if you had this on before.
[4.1] Add these programs to "Allow an app through Controlled folder access":
  • C:\Windows\SysWOW64\imdisk.exe
  • C:\Windows\System32\imdisk.exe 
  • C:\Program Files\ImDisk\ImDisk-Dlg.exe
  • C:\Program Files\ImDisk\RamDiskUI.exe
  • C:\Windows\System32\format.com
[5] Create folder, example C:\RamDiskContentBackup

This is to back up and restore RAM disk content.

Configuration

Double click on "RamDisk Configuration", and set up like this:















References