October 6, 2018

[OwnServer] Public Cloud Storage and OwnCloud (for home)

OwnCloud is like Dropbox (or Google Drive, One Drive).  This entry explains about public cloud storage and running own, using OwnCloud open source.
Public cloud storage gives some free spaces, mobile apps, and 24x7 availabilities and free of any maintenance work on users’ end.  Paid subscription gives a lot of space with reasonable price.
There are a few limitations with the public cloud storage, that can be complemented by running own cloud storage.

Public Cloud Storage

Most of people should use public storage for its convenience, free or inexpensive plan, no work is involved on user’s end.  Many services offer free storage plan and it is enough for many cases.  The paid plan also gives plenty of space for the money:
  • Google Drive: Free 15GB, 100GB - $1.99/month, 200GB – $2.99/month, 2TB – $9.99/month … and so on.
  • Dropbox: Free 5GB, 1TB – $8.25/month, 2TB – $16.58/month
  • Microsoft One Drive: Free 5GB, 50GB – $1.99/month.
  • Amazon: 100GB – Prime members get free 5GB for files/videos and unlimited space for photos.  $11.99/year, 1TB – $59.99/year on top of Amazon Prime ($119/year).
(as of 2018-10-07)

 Google Drive (new program Google One) is the best value for the money and they just added family sharing feature.  I use Google Drive and Dropbox both, occasionally use One Drive, and I am Amazon prime member but don’t use Amazon storage as I might drop Amazon prime membership.
Here is how I use each service:
  • Dropbox – this is the only public cloud client I run on most of my devices. Store files I need to share among family members and devices.
  • Google – sharing google docs among family members, and all of the smart phones upload the photos and videos automatically from phones to the cloud using Google Photos app.
  • One Drive – only for OneNote.


Problems with Public Cloud Storage

Public cloud is great.  Inexpensive, robust, and simple to use.  But there are some issues:
  • Internet Dependent - unless I synchronize all the files offline and online, if there is ISP outage, I cannot access them.
  • Privacy and Security – are we absolutely sure the provider never scan them even by any of their processes?  If they are hacked, everything will be stolen.  Dropbox, Facebook, even the credit bureau have been all hacked and data was stolen.
  • Stability, Vendor Lock-in – Once uploaded close to 1TB, most of the services I tried to start behave strangely, and finding/managing files weren’t easy.  Restoring files are not easy if you have a lot of files and taking a lot of space.  For example, after storage is filled close to 1TB, it’s not easy to find files.  And moving around the files and folder to organize were cumbersome.  Deleting didn’t work properly either for deleting large number of files.

    I tried to get off of the services by downloading them – the service started to give errors (can’t download), and took long time to get them back if successful.  So once uploaded a lot of files to the public cloud, getting them back is not easy. 

    Migrating to another service would hard – download all the files with all the troubles I described, and uploading again to new service will take weeks.

    I had some issues with cloud and this big company’s support – they weren’t very helpful, and sometimes they have no idea, or even claiming there is no issues even I described them in detail.  I lost confidence in them.

    If it’s a back up and you always have local copy, then it’s good.  You can just delete the files in cloud or delete your account.  But are we sure they will really delete them?  And, if total size of the files are less than 100GB, it would be ok – more than that, it’s not easy.

Running Own Cloud Storage

With the all of issues with public cloud, I decided to run my own cloud.  If you’re a tech-savvy person, you can run own cloud storage also.  Here are some options running own personal cloud storage:
  • Easy - External hard drive that comes with that capability (see this search on Amazon), or, get NAS storage with the feature. (search on Amazon)
  • Medium difficulty - Some routers supports this functionality, just hook up external had drive to it.  Check your router’s manual.
  • Most difficult – run open source cloud software on your own server.  This is what this blog entry is about.

Setting up Personal Cloud on Personal Server

Many families have high speed internet at home, and old computers, so setting up own cloud storage make sense to some tech-savvy folks. 
Downside of running own personal cloud on personal server are:
  • Electricity cost – it needs to run when you need to use.  I set it to run only specific time period, so I don’t have to run it 24x7.
  • Server cost – I used old computer (old core2 CPU with 4GB memory).  I just replaced booting disk to old SSD (60GB) that I wasn’t using after upgrading other computers.  So, server cost is zero.  I also already use it for other purposes, and run Raspberry Pi 24x7 for another purpose so it’s not really addition but utilizing what I already have and use.
  • Additional work as setting up and maintenance – setting it up properly for your specific needs takes time, but general set up is easy and simple: if there is only one server with no SSL and internally used only.  For maintenance, just routine normal server maintenance and checking on storage space occasionally is all.
I chose OwnCloud, since it’s stable and run on existing Apache+PHP (not a separate process) and database.  It also has apps for android, iOS, Linux, mac and windows.
Here is my network set up at home for personal cloud storage:
2018-10-06, Owncloud
There are two entry points – one from outside of home coming through Internet (see User from left side), and from the inside of the network (user at the top-right).
Some notes:
  • Router – hooked up to ISP with appropriate firewall setting and unused ports are closed.  HTTP/HTTPS ports are forwarded to RPi.
  • RPi:
    • NGINX with reverse proxy setting and named virtual host.
    • Dynamic DNS with afraid.org to register dynamic IP with a name like “mycloud.mooo.com”.  Set up custom domain name registrar’s DNS - added an entry, to point “mycloud.moo.com” to my custom domain, “owncloud.MyDomain.com”.  This is how it work:

      1. User reaches to https://ownCloud.MyDomain.com : SSL by Let’s Entrypt.  Auto updated by RPi.
      2. mycloud.mooo.com –> points to ownCloud.MyDomain.com : custom domain name registrar has the entry.
      3. mycloud.moo.com : afraid.org and dynamic IP is auto updated by RPi.
    • SSL with Let’s Encrypt
    • Runs some automation processes. 
      RPi uses very little electricity, so I run it 24x7.  I first ran everything on RPi (As storage, nginx, DB, etc), and it was a bit too much for it to handle.  So only running small processes.
  • OwnCloud: runs on old PC with Linux, Apache+PHP with named virtual host.  Since Apache/PHP is already running, other PHP based web applications are also installed and used.
  • Other servers, DB – they can be on the same server or separate.
  • Both Apache and RPi’s NGINX is configured to serve named-virtual host.  Thus it is critical to use server names.
  • All the internal servers are set to have static IPs – 192.168.1.x range, and the internal server names are added to the router’s DNS. E.g. OwnCloud running server name “oc.intranet” was added.

Setting Up NGINX

Setting up OwnCloud in Apache is simple, so I won’t go into the details here.  But with named virtual servers, SSL, it took some time to configure NGINX properly with reverse proxy.
First, add “oc.intranet” with IP to Router’s DNS.
Second, edit nginx configuration files:
$ vi /etc/nginx.conf
in “http {“ section, add this:
upstream oc-backend {
    # “oc.intranet” is added to router’s DNS
    # Apache is configured to serve named virtual server with this name, thus must use name.
     server oc.intranet:80;
}
And edit named virtual hosting file:
$ vi /etc/nginx/sites-enabled/owncloud-vhost.conf
Here is the content of the file:
# this section is for port 80 (HTTP) and tells the client to go to HTTPS instead.
server {
    listen 80;
    server_name owncloud.mydomain.com;
    return 301 https://$server_name$request_uri;
}
# SSL (HTTPS) section
server {
    server_name owncloud.mydomain.com;
    # only if you have country blocking enabled (GeoIP)
    if ($allowed_country = no) {
        return 444;
    }
    # SSL port
    listen 443 ssl;
    listen [::]:443 ssl;
    ssl on;    # let’sencrypt cert section
    ssl_certificate /etc/letsencrypt/live/owncloud.mydomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/owncloud.mydomain.com/privkey.pem;

    ssl_protocols  SSLv2 SSLv3 TLSv1;
    ssl_ciphers  HIGH:!aNULL:!MD5;
    location ~ /.well-known {
        allow all;
    }
    access_log /vhost/sites/owncloud/logs/access.log main;
    error_log  /vhost/sites/owncloud/logs/error.log;
   
    location / {
        # reverse proxy section
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Real-IP $remote_addr;

        # optional:
        proxy_buffering off;
        proxy_connect_timeout 300;

        # below is set up in nginx.conf file, this forces to use “name” for apache
        # if I put the host name, “oc.intranet”, even the name is in router’s DNS,
        # it’s not resolved at start up time  Thus use upstream section in nginx.conf
        # and refer that from here.
        proxy_pass http://oc-backend/;

        add_header Front-End-Https on;
        proxy_set_header Connection "Keep-Alive";

        # reverse proxy logging, optional:
        access_log  /vhost/sites/owncloud/logs/proxy-access.log upstreamlog;    }
    client_max_body_size 1024m;
}

The highlighted lines are important for settings like mine:
https://CustomDomain (nginx, named virtual host) –> http://internalDomain (apache, named virtual host)
OwnCloud can be accessed over the internet (https://owncloud.mydomain.com), and also within the network (http://oc.intranet).  Don’t add BAISC auth in RPi layer – OwnCloud itself uses BASIC auth and it interferes and login will not work. I spent hours to figured this out.
For security  following should be used:
  • fail2ban
  • hide server info, customize error page.
  • Country based block – this reduces a lot of automated hacking attempts.
  • Change SSH port number – also reduces a lot of attempts.
  • Close all other ports.
  • The best way to be safe, don’t open the port and don’t make OwnCloud server accessible to outside.

Conclusion

I did this because I already run Linux servers (using old PCs and Raspberry Pi) for web server, file server, database, git server, jenkins, search server, SonarQube, coding for linux, samba, etc.  I also have 100/100Mbps internet connection, so it’s good enough to run as a server for family use.
Other reasons are that I have a lot of files I want to put on “cloud”, and may be some sensitive files, and want to access the files even IPS have issues.  I can do sneaker-net to “upload” or retrieve large number of files fast. 
If you don’t have a Linux server already running, use public cloud – and store less than 100GB files and don’t put any sensitive files.  If you decided to set up own personal cloud for storage only, get a NAS. 
So only run your own cloud like OwnCloud, if you already have Linux server, and don’t want to buy NAS, or doing this for fun. 

References

No comments:
Labels: , ,
Subscribe to: Posts (Atom)