Friday 25 July 2014

Removing hosts from backuppc

Simply remove the host from the web interface and rm -rf the pc/<host> directory, then wait for the next BackupPC_nightly run - it will remove all superfluous files from the pools.

The path for this directory usually is:
/var/lib/Backuppc/pc/<hostname>
If you want to force the clean-up process, you can remove your host like this:

1. Login to the Backuppc server
2. Remove the host in the Backuppc web-interface (under hosts)
3. remove it's directory /var/lib/Backuppc/pc/<hostname>:
rm -rf /var/lib/Backuppc/pc/<hostname>
4. Shutdown backuppc:
service backuppc stop
5. Change into backuppc:
su - backuppc
6. Run the nightly script:
/usr/share/BackupPC/bin/BackupPC_nightly 0 255
7. Go back to root:
exit
8. Start backuppc again:
service backuppc start

Possibly Related Posts

Reclaim free space from Time Machine sparsebundle

You msut run these commands as root:
sudo su - 
Make sure the mount point exists:
mkdir -p /Volumes/TM
Then mount the afp share:
mount_afp 'afp://user:password@afp_server_address/share_name' /Volumes/TM
Now use hdiutil to reclaim the available free space:
hdiutil compact /Volumes/TM/ComputerName.sparsebundle/
unmont the share:
umount /Volumes/TM/
If you get an error message saying:
hdiutil: compact failed - Resource temporarily unavailable
You must make sure you don't have the afp share mounted elsewhere, you can check your mounts with:
df -h
If the output contains a line with your afp server's address or with the string "Time Machine" you have to unmount them.

The following script will do all that for you:
SRV="afp_server_address"
SAVEIFS=$IFS
IFS=$'\n';
for v in $(df -h | grep -E "$SRV|Time\sMachine"  | cut -d"%" -f3 |  sed -e "s/ *\//\//"); do
    umount "$v"
done
IFS=$SAVEIFS
mkdir -p /Volumes/TM
mount_afp "afp://user:password@$SRV/share" /Volumes/TM
hdiutil compact "/Volumes/TM/$(scutil --get ComputerName).sparsebundle"
umount /Volumes/TM/

Possibly Related Posts

Thursday 24 July 2014

Changing Time Machine Backup Interval

You can use the following command to change the Time Machine backup interval:
sudo defaults write /System/Library/LaunchDaemons/com.apple.backupd-auto StartInterval -int 14400
The time interval is in seconds, so 43200 will start a backup every 12hrs.

Checkout my previous post to learn how to manually delete Time Machine backups.

Possibly Related Posts

Manage time machine backups

Some times you get some errors saying that your time machine's disk is full and Time Machine could not complete the backup so you need to  manually delete old backups.

tmutil provides methods of controlling and interacting with Time Machine, as well as examining and manipulating Time Machine backups. Common abilities include restoring data from backups, editing exclusions, and comparing backups.
tmutil latestbackup
Will output the path to the most recent backup and
tmutil listbackups
will list all existing backups, if you use the same backup disk for multiple machines, you can get just the backups from your machine with:
tmutil listbackups | grep "$(scutil --get ComputerName)"
The following command will delete the backups from a mac named old_mac_name:
sudo tmutil delete /Volumes/drive_name/Backups.backupdb/old_mac_name
If you want to be safe, you can pick one snapshot to delete first to be sure the command works as intended. This is nice since it could take hours to clean up some larger backup sets and you want to leave the Mac confident it's deleting the correct information store.

You can use the tmutil tool to delete backups one by one.
sudo tmutil delete /Volumes/drive_name/Backups.backupdb/mac_name/YYYY-MM-DD-hhmmss
Since tmutil was introduced with Lion, this will not work on earlier OS versions.

The tmutil delete command only removes the backup from the sparse bundle. It doesn’t actually free the disk space. To do that, you have to go a little deeper.

On your Mac is a mount point called /Volumes. You can examine the contents of this mount point with ls:
cd /Volumes
ls -1
Should output something like:
Macintosh HD
Recovery HD
Time Machine Backups
TimeMachine
These are the names of all the mounted disks (or things that look like disks) on your Mac. Notice two likely candidates for your actual TimeMachine volume. Yours may be named slightly differently, but the one you want is the one that actually shows files of type .sparsebundle . In my case, it is the volume TimeMachine:
sudo ls -l TimeMachine/
and you should see something similar to:
...
drwxr-x---@ 1 root wheel 264 Jul 25 08:21 sysadmin’s MacbookPro.sparsebundle
...
Notice that you don’t actually own the file. (Had I not used the sudo command with ls I could not have listed the contents of /Volumes/TimeMachine)

That .sparsebundle file for your Mac is where all your backup sets live. TimeMachine manages the contents of this file, but doesn’t do anything automatically to reduce its size. Luckily there is another tool for that, but you’ll have to be root to run it:
sudo su -
hdiutil compact /Volumes/TimeMachine/YourBackup.sparsebundle
Sample output:
Starting to compact…
Reclaiming free space…
...................................................
Finishing compaction…
Reclaimed 3.1 GB out of 304.1 GB possible.
That’s it! In this example I reclaimed 3.1GB of actual disk space on my TimeMachine volume. 

The following bash script will remove the oldest backup and reclaim the free space:
COMPUTER_NAME=$(/usr/sbin/scutil --get ComputerName)
NBACKUPS=$(/usr/bin/tmutil listbackups | /usr/bin/grep "$COMPUTER_NAME" | /usr/bin/wc -l)
OLDEST_BACKUP=$(/usr/bin/tmutil listbackups | /usr/bin/grep "$COMPUTER_NAME" | /usr/bin/head -n1)
LATEST_BACKUP=$(/usr/bin/tmutil latestbackup)
echo Latest backup: $LATEST_BACKUP
if [[ -n "$LATEST_BACKUP" && "$LATEST_BACKUP" != "$OLDEST_BACKUP" ]]
then
    echo "$NBACKUPS backups. Delete oldest: ${OLDEST_BACKUP##*/} [y/N]? \c"
    read answer
    case $answer in
   y*)
  echo Running: /usr/bin/sudo /usr/bin/tmutil delete "$OLDEST_BACKUP"
  /usr/bin/sudo time /usr/bin/tmutil delete "$OLDEST_BACKUP"
    echo "Do you wish to reclaim the free space now? [y/N]? \c"
    read answer
    case $answer in
    y*)
     mkdir -p /Volumes/TM
     mount_afp 'afp://user:pass@afp_server_address/share_name' /Volumes/TM
     hdiutil compact "/Volumes/TM/$(scutil --get ComputerName).sparsebundle"
     umount /Volumes/TM/
    ;;
    *)
 echo No change
    ;;
        esac
   ;;
   *)
  echo No change
   ;;
    esac
else
    echo "No backup available for deletion"
fi
In the script above, don't forget to change the afp URL (afp://user:pass@afp_server_address/share_name) to your own.

Possibly Related Posts

Wednesday 23 July 2014

Sorry, Command-not-found Has Crashed

When you try to execute a command that is not installed Ubuntu tries to hint you on the package that you should install but some times, especially after an upgrade, you get an error message saying:
Sorry, command-not-found has crashed! Please file a bug report at:
(...)
This solves the problem:
export LANGUAGE=en_US.UTF-8
export LANG=en_US.UTF-8
export LC_ALL=en_US.UTF-8
locale-gen en_US.UTF-8
sudo dpkg-reconfigure locales

Possibly Related Posts

Ubuntu as Time Machine server

This guide will help you to install and configure the netatalk servise on an Ubuntu server so it can function as a Time Machine backup server for your Mac OS machines.

First install the necessary packages:
sudo apt-get install netatalk avahi-daemon libnss-mdns
Open the netatalk default configuration file:
sudo vi /etc/default/netatalk
Modify the lines:
ATALKD_RUN=yes
PAPD_RUN=no
CNID_METAD_RUN=yes
AFPD_RUN=yes
TIMELORD_RUN=no
A2BOOT_RUN=no
Edit the atalkd.conf file:
sudo vi /etc/netatalk/atalkd.conf
add to the bottom
eth0
Edit the AppleVolumes.default file:
sudo vi /etc/netatalk/AppleVolumes.default
add to the bottom:
/backups/timemachine "Time Machine" allow:@admin cnidscheme:cdb volsizelimit:200000 options:usedots,upriv,tm
The example above also limits the size shown to OS X as 200 GB (the number is given in MiB, so it's 200,000 times 1024 in the real world)

Edit the afpd configuration file:
sudo vi  /etc/netatalk/afpd.conf
add to the bottom:
- -transall -uamlist uams_dhx.so,uams_dhx2.so -nosavepassword -advertise_ssh -mimicmodel TimeCapsule6,106 -setuplog "default log_warn  /var/log/afpd.log"

Create a configuration file for the avahi afpd discovery:
sudo vi  /etc/avahi/services/afpd.service
and enter the following into it:
<?xml version="1.0" standalone='no'?><!--*-nxml-*-->
<!DOCTYPE service-group SYSTEM "avahi-service.dtd">
<service-group>
 <name replace-wildcards="yes">%h</name>
 <service>
 <type>_afpovertcp._tcp</type>
 <port>548</port>
 </service>
 <service>
 <type>_device-info._tcp</type>
 <port>0</port>
 <txt-record>model=MacPro</txt-record>
 </service>
</service-group> 
Restart the services:
sudo service netatalk restart
sudo service avahi-daemon restart

Possibly Related Posts

Saturday 19 July 2014

MySQL and Oracle command equivalents

MySQL has specific commands, which provides the easy access to the information_schema database, to get the schema level details. But oracle does not provide such easy access to some of the schema level meta data.

Here are  some MySQL specific commands/Syntaxes & equivalent Oracle techniques:

To get the list of databases

MySQL :
show databases
Oracle :
SELECT username FROM all_users ORDER BY username;

To get the current schema

MySQL :
select DATABASE();
Oracle :
SELECT sys_context('USERENV', 'CURRENT_SCHEMA') FROM dual;

To get the list of tables within the current database

MySQL :
use database_name;
show tables;
Oracle :
select * from user_tables;
Here schema is based on the connected username, so it is selected during the creation of the connection.
USER_TABLES will have a row for every table in your schema. If you are looking for the tables in your schema, this would be the correct query. If you are looking for the tables in some other schema, this is not the right table to use.

ALL_TABLES will have a row for every table you have access to regardless of schema. You would, presumably, want to qualify the query by specifying the name of the schema you are interested in, i.e.
SELECT table_name
  FROM all_tables
 WHERE owner = <<name of schema>>
Of course, that assumes that you have at least SELECT access on every table in that schema. If that is not the case, then you would need to use DBA_TABLES (which would require that the DBA grant you access to that table), i.e.
SELECT table_name
  FROM dba_tables
 WHERE owner = <<name of schema>>

To get the connected connection info

MySQL :
show processlist
Oracle :
SELECT sess.process, sess.status, sess.username, sess.schemaname, sql.sql_text FROM v$session sess, v$sql sql WHERE sql.sql_id(+) = sess.sql_id AND sess.type = 'USER';

To limit the selection

MySQL :
select * from user limit 10;
Oracle :
select * from table_name where ROWNUM <= 10;
To select rows which is somewhere middle

MySQL :
select username from user limit 10, 15;
Oracle :
select element_name from (select element_name, ROWNUM as row_number from table_name) as t1 where t1.row_number > 10 and t1.row_number <= 15; 
Note: Here we have to use SubQuery rather than call it directly as "select element_name from table_name as t1 where ROWNUM > 10 and ROWNUM <= 15;". This cannot be done as these ROWNUMs are assigned once they are satisfied the given conditions, which follows the WHERE. Since condition "ROWNUM > 10 and ROWNUM <= 15" will never be satisfied from the start ROWNUMs will never be incremented. So we need to use the Subqueries to let the ROWNUMs assigned within the Subquery and later filter the required results from the outside query.

Describe table has a same syntax in both MySQL & Oracle.
desc table_name;
To view errors/warnings

MySQL :
show warings / show errors
Oracle :
select * from user_errors;/ show errors
MySQL has auto_increment Columns

MySQL :
create table table_name (element_id int AUTO_INCREMENT primary, element_name varchar(20));
Oracle :
i) Create table without the auto_increment keywords (because it does not exist in Oracle)
create table table_name (element_id int primary, element_name varchar(20));
ii) Create a sequence, which provides the incremented values
create sequence auto_incrementor;
iii) Create a trigger, which gets the next value from the sequence and updates it to the column to be auto_incremented
CREATE TRIGGER trig_incrementor BEFORE INSERT ON table_name
FOR EACH ROW
BEGIN
SELECT auto_incrementer.NEXTVAL into :new.element_id FROM dual;
END;
To get the table create script back

MySQL :
show create table table_name;
Oracle :
Make sure that the select_catalog_role is already available for the given user if not assign the role, as shown below.
grant select_catalog_role to [username];Increase the page size and maximum width for displaying the results so that complete table definition can be displayed in the sqlplus console.
set pagesize 999
set long 9000
select dbms_metadata.get_ddl('TABLE', 'TABLE_NAME', 'database_name') from dual;
To get the session variables

MySQL :
show variables; or show variables like 'inno%';
Oracle :
SELECT name, value FROM gv$parameter; or SELECT sys_context('USERENV', ) FROM dual;

Explain the execution plan of a sql statement

MySQL :
explain select * from table_name;
Oracle :

i) First execute the explain plan so that it will fill the plan_table (this table need to be created according to the standard plan_table format, if it does not exist already)
explain plan select * from table_name;
ii) Now the results of the explain plan will be populated in the plan_table, so we need to use a row connecting query to get a readable summary of the results.
select substr (lpad(' ', level-1) || operation || ' (' || options || ')',1,30 ) "Operation", object_name "Object" from plan_table start with id = 0 connect by prior id=parent_id;

Possibly Related Posts

Thursday 17 July 2014

MySQL and Postgres command equivalents

Task: list existing databases, connect to one of the databases, then list existing tables and finally show the structure of one of the tables.

In Mysql:
show databases;
use database;
show tables;
describe table;
exit
In PostgreSQL:
\l
\c database
\dt
\d+ table
\q

Possibly Related Posts

Wednesday 16 July 2014

Test imap using telnet

For added security, you can encrypt your IMAP connection. This requires that your server supports SSL or TLS and that you have access to an SSL/TLS client program, for example OpenSSL, to use instead of telnet.

As the port-number normally is 993, an example OpenSSL command would be openssl s_client -connect imap.example.com:993 -quiet. (If you would like to see the public key of the server, as well as some other encryption-related information, omit -quiet.) The server should then start an IMAP session, displaying a greeting such as the * OK Dovecot ready example below.
telnet imap.example.com 143

#output: Trying 193.136.28.29...
#output: Connected to imap.example.com.
#output: Escape character is '^]'.
#output: * OK Dovecot ready.

a1 LOGIN MyUsername MyPassword
#output: a1 OK Logged in.

a2 LIST "" "*"
#output: * LIST (\HasNoChildren) "." "INBOX"
#output: a2 OK List completed.

a3 EXAMINE INBOX
#output: * FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
#output: * OK [PERMANENTFLAGS ()] Read-only mailbox.
#output: * 1 EXISTS
#output: * 1 RECENT
#output: * OK [UNSEEN 1] First unseen.
#output: * OK [UIDVALIDITY 1257842737] UIDs valid
#output: * OK [UIDNEXT 2] Predicted next UID
#output: a3 OK [READ-ONLY] Select completed.

a4 FETCH 1 BODY[]
#output: * 1 FETCH (BODY[] {405}
#output: Return-Path: sender@example.com
#output: Received: from client.example.com ([192.0.2.1])
#output:    by mx1.example.com with ESMTP
#output:    id <20040120203404.CCCC18555.mx1.example.com@client.example.com>
#output:    for <recipient@example.com>; Tue, 20 Jan 2004 22:34:24 +0200
#output: From: sender@example.com
#output: Subject: Test message
#output: To: recipient@example.com
#output: Message-Id: <20040120203404.CCCC18555.mx1.example.com@client.example.com>
#output:
#output: This is a test message.
#output: )
#output: a4 OK Fetch completed.

a5 LOGOUT
#output: * BYE Logging out
#output: a5 OK Logout completed.

Possibly Related Posts

Source IP based reverse proxy

If you want to proxy some source IP ranges/subnets to one server and onther subnets to go to a different server, you can do this using mod_rewrite for proxying. You will have to setup a rewrite condition based on the source IP and a rewrite rule with the [P] flag. Something like this should work:
RewriteCond %{REMOTE_ADDR} ^10\.2\.
RewriteRule ^/(.*) http://old-app/$1 [P]
ProxyPassReverse / http://serverA/

RewriteCond %{REMOTE_ADDR} ^10\.3\.
RewriteRule ^/(.*) http://new-app/$1 [P]
ProxyPassReverse / http://serverB/
One possible scenario for this if you want to migrate your users from server A to server B but you want to migrate one IP range at the time.

If you're using Apache 2.4 or newer you can also achieve this with the following configuration:
<If "-R '10.2.0.0/16'">
  ProxyPassReverse / /http://serverA/
</If>
<ElseIf "-R '10.3.0.0/16'">
  ProxyPassReverse / /http://serverB/
</ElseIf>
<Else>
  ProxyPassReverse / /http://serverC/
</Else>

Possibly Related Posts

Tuesday 15 July 2014

Preserving client ip with apache reverse proxy

The first thing that I thought of was the “X-Forwarded-For” headers, which is an HTTP header inserted into the original HTTP GET request whose value is equal to the client’s public IP. Turns out apache reverse proxy inserts this header by default. So we somehow need to instruct the backend server itself to provide the application with the correct client IP.

If your backend server is a Tomcat server the solution cam be using the RemoteIP tomcat valve.

It’s quite simple to configure in that all that needs to be done is to modify tomcat server.xml to recognise original client IP rather than the proxy IP by adding the following to server.xml:
<Valve className="org.apache.catalina.valves.RemoteIpValve" internalProxies="127\.0\.0\.1" />
make sure to change 127.0.0.1 to the address of the apache reverse proxy.

The application could now recognise the original client IP.

The apache equivalent of the above method is using mod_rpaf for Apache 1.3 & 2.2.x and mod_remoteip for Apache 2.4 and 2.5. 

These apache modules can be used to preserve both remote IP/HOST. Internally they use X-Forwarded-For header to detect a proxy in it’s list of known proxies and reset the headers accordingly. This works with any proxy server in the front end provided that the proxy server sets X-Forwarded-For header. 

To use mod_rpaf, install and enable it in the backend server and add following directives in the module’s configuration:
RPAFenable On
RPAFsethostname On
RPAFproxy_ips 127.0.0.1
Remote IP is automatically preserved when RPAFenable On directive is used. RPAFsethostname On directive should be used to preserve host and RPAFproxy_ips is the list of known proxy ips.

Restart backend apache server and you are good to go.

For mod_remoteip, it’s a bit similar, the configuration should look something lke this:
RemoteIPHeader X-Real-IP
RemoteIPInternalProxy 1.2.3.4
RemoteIPInternalProxy 5.6.7.8
mod_remoteip however has a lot more configuration options.

When the proxy server is an Apache, ProxyPreserveHost directive in mod_proxy can be used to preserve the remote host not the remote ip. This is useful for situations where name based virtual hosting is used and the backend server needs to know the virtual name of host.
Open mod_proxy configuration file of your proxy server and enter directive, ProxyPreserveHost On, and restart your apache instance.

Possibly Related Posts

Thursday 10 July 2014

Finding external IP using the command line

The easiest way is to use an external service via a commandline browser or download tool. Since wget is available by default in Ubuntu, we can use that.

To find your ip, use:
wget -qO- http://ipecho.net/plain ;
You can do the same using curl:
curl ipecho.net/plain ; echo

Possibly Related Posts

Wednesday 9 July 2014

How to test a listening TCP/UDP port through nc

Netcat (nc) can also be used for a lot of other purposes. It can also be used as a very fast basic port scanner, you can scan a port or a range.

To scan a range of UDP ports 1 to 1000:
nc -vzu destination_ip 1-1000
To scan a range of TCP ports 1 to 1000
nc -vz destination_ip 1-1000

Possibly Related Posts