September 2, 2008

Editing large text files

Ok, straight to the point: there is NO editor that can handle huge text file, size of giga bytes.  I have this SQL file, 10+ GB, and no editor can handle it.

One commercial software I found is: http://www.textmaster.ca/

It claims it can handle 300GB file.  It's nice, but I don't do this too often, so sorry, I don't want to purchase one.  One easy way to go around the problem is, of course, splitting the file into multiple files and edit each, and later join the files together.

Split:
[sourcecode language='css']
split -b 50m example.sql example_
[/sourcecode]
Join:
[sourcecode language='css']
cat example_[aa-bb] > example.sql
[/sourcecode]
This will do.

Back up using DOS batch

Almost always I make backups of important files, but I still lose up-to-date files.  So automating/scheduling backup is needed.

I do:

  • Files are backed up continously using Mozy.

  • Image of C drive is created daily onto another HDD.

  • Some important files are backed up using SyncBack to another computer.

  • For MySQL, I use MySQL Administrator to schedule back up.


Fo sync back, it is easier if a directory is ZIPed up.  So here is a simple batch file:

[sourcecode language='css']

@echo off
REM =============
REM back up daily
REM Run by WinXP scheduler
REM =============

REM === Get current date
FOR /F "TOKENS=1* DELIMS= " %%A IN ('DATE/T') DO SET CDATE=%%B
FOR /F "TOKENS=1,2 eol=/ DELIMS=/ " %%A IN ('DATE/T') DO SET mm=%%B
FOR /F "TOKENS=1,2 DELIMS=/ eol=/" %%A IN ('echo %CDATE%') DO SET dd=%%B
FOR /F "TOKENS=2,3 DELIMS=/ " %%A IN ('echo %CDATE%') DO SET yyyy=%%B
SET date=%yyyy%%mm%%dd%

REM KK-20080829: backing up Palm
zip -r "C:\BACKUP\palm\kkim_%date%.zip" "C:\Program Files\palmOne\kkim"

[/sourcecode]

I got this "date" batch script from the net.  Sorry, don't remember where I got it.  This works very well, and kudos to who wrote it.

I have cygwin installed and c:\cygwin\bin is in my PATH, thus the 'zip' executable is used in the batch file.  And Windows scheduler is used here.  And, SyncBack will pick up the file and put it onto another computer.  Or, the ZIP file can be created on another HDD.