Nejko's randomness » Technology

phpBB: Export all posts for a user into a file

admin — Sun, 13 Feb 2011 23:58:46 +0000

Few days ago, one of the users of the phpBB-based forum I administer, asked if he could have all his posts (few hundreds of them) exported as text and delivered via e-mail. After a quick Google search, I could find any such script, so I created my own. I wrote it in Python and the only requirement is that you have MySQLdb Python module installed (python-mysqldb in Debian).

Here it goes (I am a Python beginner, so beware):

#!/usr/bin/python

import MySQLdb
import time
import re
import getopt
import sys

def usage():
print 'Usage: PhpbbExportPosts.py [options]'
print
print 'Obligatory arguments:'
print
print ' DBHOST: MySQL database hostname'
print ' DBUSER: MySQL database user\'s username'
print ' DBPASS: MySQL database user\'s password'
print ' DBNAME: MySQL database name of phpBB'
print ' POSTER: Username of the author of the posts to be exported'
print
print 'Options:'
print
print ' --prefix=PREFIX\tTable name prefix (default: phpbb_)'
print ' --file=FILE\t\tOutput file name (default: none (stdout))'
print ' --help\t\tPrint this help'

def main():
# Finish at once if there are to few arguments
if (len(sys.argv) < 6):
usage()
sys.exit(2)

# (Try to) parse the options
try:
opts, args = getopt.getopt(sys.argv[6:], 'pfh', ['prefix=', 'file=', 'help'])
except getopt.GetoptError, err:
print str(err)
usage()
sys.exit(2)

# Set options defaults
file = None
prefix = 'phpbb_'

# Set the variables regarding to the options
for o, a in opts:
if o in ('-h', '--help'):
usage()
sys.exit()
elif o in ('-p', '--prefix'):
prefix = a
elif o in ('-f', '--file'):
file = a
else:
assert False, "Unhandled option"

# Establish connection to MySQL
conn=MySQLdb.connect(host=sys.argv[1],user=sys.argv[2], \
passwd=sys.argv[3],db=sys.argv[4])

# Set character set to UTF-8, which should be correct for phpBB
conn.set_character_set('utf8')

# Get a database cursor and set UTF-8 everywhere where possible
cursor = conn.cursor()
cursor.execute("SET NAMES utf8;")
cursor.execute("SET CHARACTER SET utf8;")
cursor.execute("SET character_set_connection=utf8;")

# Run the MySQL query to get all the posts of the selected poster
cursor.execute("SELECT a.post_time, a.post_subject, a.post_text \
FROM " + prefix + "posts a, " + prefix + "users b \
WHERE a.poster_id=b.user_id \
AND b.username='" + sys.argv[5] + "'")

# Open file for writing or use standard output
if file is not None:
output=open(file, 'w')
else:
output=sys.stdout

# Print the formatted posts into a file one by one
while (1):
# Get the next post
row = cursor.fetchone()

# Exit when there are no more posts
if row == None:
break

# Inter-post delimiter line
output.write('================================================================================\n')

# Also remove all HTML tags with a regular expression
output.write('Post date: %s\nPost subject: %s\nPost content:\n\n%s\n\n' % (time.ctime(row[0]), row[1], re.sub('<.*>', '', row[2])))

print "Number of posts exported: %d" % cursor.rowcount

if __name__ == "__main__":
main()

Alternatively, you can download it from here.

IPv6 address database: ipv6list.com

admin — Fri, 11 Feb 2011 23:10:29 +0000

A few weeks ago I saw a presentation from van Hauser titled Recent advances in IPv6 insecurities. He merges three different sources of IPv6 addresses to obtain a database of IPv6 addresses:

search engines and databases (directories),
DNS (bruteforcing),
common addresses.

He shows that this method is quite successful for those who are interested in getting as many IPv6 addresses as possible. So I was thinking – why not put up a public databases of such addresses and provide a way for anybody to contribute them.

The main purpose of such database is to make IPv6 Internet a bit more “enumeratable” for those who would like doing any kind of research on the nature and behaviour of IPv6 Internet. Also, it may convince those who would like to not be listed (I guess for security through obscurity reasons) to actually renumber from ::1 to something else.

Of course, any ideas about other possible techniques for harvesting IPv6 addresses, are welcome.

netfilter’s way of tracking ICMPv6 connections

admin — Fri, 11 Feb 2011 02:06:56 +0000

These days I am configuring a Debian-based firewall, whose iptables policy is being managed by Firewall Builder tool. Check it out, it’s a really cool, object-based firewall policy management tool.

However, I was quite stumped when I realized that if I make an “allow all” mixed (IPv4/IPv6) policy, the ICMPv6 packets are not let through the bridge (I am doing filtering on a bridge interface). This is not really intuitive, so I thought it must be some kind of bug. I still recall, that some time ago, netfilter had serious problems tracking IPv6 connections (it would match improperly match valid connections, which made it impossible to drop the packets, which matched INVALID state).

My rationale was: I should pass all IPv6 packets, which match NEW, RELATED or ESTABLISHED state in both directions and I should be fine. This is also what Firewall Builder generates. However, I found out that ICMPv6 packets other than PING (echo/reply) don’t match netfilter states at all! So that was why Firewall Builder’s policy wouldn’t let my ICMPv6 packets (e.g. neighbor discovery) through.

The solution is straightforward: make an additional “allow any” rule only for ICMPv6 packets and make it stateless. Firewall Builder nicely supports that, you can double click the Options column and check the “Stateless” check box.

SharePoint Services 3.0 dead after applying KB983444

admin — Wed, 13 Oct 2010 12:01:14 +0000

I am not a Windows administrator, really. But I do take care of one Windows 2008 SBS machine and we have Exchange 2007 and SharePoint Services 3.0 installed. Recently, I installed a security update for Windows SharePoint Services 3.0 (KB983444). After rebooting, all my SharePoint sites were down (404 error). It took me quite a few hours to put them back online. I did tried many things, among them:

running SharePoint Products and Technologies Configuration Wizard, which failed at step 9/10,
running “psconfig -cmd upgrade -inplace b2b -wait -force“, which failed with various errors.

Finally, I did this to solve the problem: I connected to Central Administration and then went to “Operations” tab. Then, I chose “Services on Server” and clicked “Windows SharePoint Services Search”. There, I entered my administrator account as the “Service Account”. I saved the changes and then rerun SharePoint Products and Technologies Configuration Wizard, which worked this time (although it took it very long time to do everything it had to do).

RCPFA 1.0.5 available

admin — Sat, 31 Jan 2009 23:05:15 +0000

The 1.0.5 version is out. It makes RCPFA compatible with RoundCube 0.2-stable. Thanks to Andrey Sharandakov!

gmirror and gvinum on the same drives

admin — Sun, 30 Nov 2008 21:05:30 +0000

In 2006, when I was installing a FreeBSD server for our client, one of the requests was also a RAID-5 array of some kind. I checked out and discovered GEOM vinum (or gvinum), which provided what I needed at that time. It is a file server, but throughput is not a critical issue, so I tried it (at that time, graid5 was not yet available, AFAIK). I am writing this because this weekend I had to rebuild the array (and copy the data) with new, larger drives, which took me many hours to do it, because there is not so many documentation on how to make different GEOM RAID subsystems share the same three drives.

This is what I wanted to achieve: have three drives, which would contain two gmirror (RAID-1) arrays (one for root partition, the other for swap) and three gvinum (RAID-5) volumes – for

/var

/tmp

and

/usr

. For the latter, it is best to use volume management capabilities of gvinum, which allows you to join only three physical devices (or slices or partitions) with it, so that the logical volumes are created “inside” the vinum manager.

The main problem was, that I forgot how to do this “properly”. It was 2 years since I did this (of course, I didn’t write it down how I did it, although it took me a few hours) last time and since the machine is far away, I don’t have physical access. This would have helped, because I could just put the old drives back and see how they were configured, but the remote system administrator already exchanged the drives and I didn’t want to bother him.

In FreeBSD terms, a partition is a logical unit, which resides on a slice (which is actually a partition from other operating systems’ point of view). Let’s have four drives on the system:

/dev/ad0

/dev/ad1

/dev/ad2

and

/dev/ad3

. We’ll assume that on

/dev/ad0

there is the system we are booting and running FreeBSD at the moment and we wan’t to create the arrays on the other three drives, which will eventually run by themselves (we’ll pull the

/dev/ad0

out when we finish). When you create a slice on

/dev/ad1

, for example, you’ll be able to access it via

/dev/ad1s1

. When you create a partition on this slice, you’ll see it as

/dev/ad1s1a

, where the last letter “a” can also be “b”, “d”, “e”, and so on, you know the alphabet. This naming system is somewhat peculiar, and I don’t like it but I can live with it. The letter “a” usually hosts the root partition, and the letter “b” provides swap space. As you can see, there is no letter “c”. This is because it specifies the whole slice and therefore it should not be used for anything else.

Usually, when you’re setting up the gmirror RAID-1 on FreeBSD, you make put it on the physical drive directly, i.e. you make the

/dev/ad0

visible as

/dev/mirror/gm0

(after you put the metadata on the drive, by running ‘

gmirror create

‘), which also means that all existing slices and partitions will be visible at the new location. If you had

/dev/ad0s1a

, you’ll now have

/dev/mirror/gm0s1a

. Which is very nice and makes gmirror very easy to set up after the system was installed. In the end, you just add other mirrors (

/dev/ad1

, …) in the array and that’s it.

However, if you want to use gvinum on the same drives (to make RAID-5 arrays, for example), you can’t do that. You’ll need to gmirror something else: the slices or the partitions, but not the whole drives. Now FreeBSD allows you to have no slices at all – to create the FreeBSD partitions (the letters) directly on the drive (so you’ll have

/dev/ad1a

). So when I started to think about how I would partition the drives and which units will I merge with gvinum and gmirror, I became a bit confused. So I tried a few ideas I had and none of them really worked because I didn’t know what actually the command such as ”

bsdlabel -w

“, ”

boot0cfg

“, ”

gmirror label

“, ”

gvinum create

” and creating slices via ”

sysinstall

” actually do. Where do they write their data? At what offsets and what are the sizes of these metadata? I found it quite annoying that there isn’t much documentation about this (at least not well organized), so I had to do some homework. Here is what I discovered:

gvinum — When you run ”

gvinum create

“, it will rewrite the bytes from

0x1000

0x21200

, that is from block 8 (first 8 blocks are left untouched) to block 265 with its own configuration data, so you have to be careful not to mess with these blocks.

gmirror — Running ”

gmirror label

” puts gmirror’s metadata on the last block of the device. The size in blocks of the mirror is then number of block of the device – 1.

bsdlabel — When labelling a slice (or the drive directly), bsdlabel writes label information to the second block (from address

0x200

on, in my tests it never passed the

0x2c0

limit, which still fits into the second block).

boot0cfg — Since it rewrites the MBR with BootMgr, this means it rewrites the first block (block 0) of the drive.

fdisk — Fdisk writes the slice information into first 16 blocks of the slice. This means, that you shouldn’t label them with bsdlabel (don’t assign them to any of the partitions), or you can have problems.

To sum up, the only configuration, which worked for me on FreeBSD 6.1 (yes, quite old one) was the following. First I created slices on all of the drives (one on each drive) and wrote the BootMgr onto them (you can do this easily by running

sysinstall

and then going to Custom and then Partition. You select the first drive (of the three) and then, when in fdisk-editor, press a and then w to write the slice. When asked about MBR, just say BootMgr and that’s it. This will ensure that there is a boot manager on the drive (which means you can boot from it). You have to repeat this procedure for the other (two) drives as well.

Then, you have to edit the label of all three slices, running “

bsdlabel -e /dev/ad1s1

” (for the slice on the first drive). You have to provide the following partition set:

a:  1048576       16    4.2BSD        0     0     0
b:  4194304  1048592      swap
c: 976768002       0    unused        0     0         # "raw" part, don't edit
d: 971525106 5242896     vinum

In this configuration you can see that the size of “a” (root) partition is 1048576 512-byte blocks which means 512 MB. The offset of 16 blocks for the “a” partition is very important, since the slice needs the first 16 blocks for itself. The size of the “b” (swap) partition is 4 times the size of “a” (2 GB) and the “d” takes all the space left on the slice.

So the idea is to make two gmirror arrays, the first one will consist of the three “a” partitions (remember, we have three hard drives) and will be used as the root partition. The second one will consist of the three “b” partitions and will be used as swap space. All the “d” partitions will be used for the construction of the gvinum array.

First, you need to load the

geom_mirror module

, which enables kernel to handle the gmirror arrays. You do this by running ”

kldload geom_mirror

“. But, it is needed to make this change permanent (so the module will load at boot) so you need to add these two lines to

/boot/loader.conf

geom_mirror_load="YES"
geom_vinum_load="YES"

This will also enable loading gvinum at boot, which we will need later (when the system will boot from the new arrays). No it’s time to create the arrays. You’ll run something like:

# gmirror label -v -b round-robin root /dev/ad1s1a
# gmirror label -v -b round-robin swap /dev/ad1s1b
# gmirror insert root /dev/ad2s1a
# gmirror insert root /dev/ad3s1a
# gmirror insert swap /dev/ad2s1b
# gmirror insert swap /dev/ad3s1b

This was for the gmirror arrays. Now make a file named

gvinum.conf

and put this in it:

drive disk1 device /dev/ad1s1d
drive disk2 device /dev/ad2s1d
drive disk3 device /dev/ad3s1d
 volume var
  plex org raid5 491k
   sd length 1024m drive disk1
   sd length 1024m drive disk2
   sd length 1024m drive disk3
 volume tmp
  plex org raid5 491k
   sd length 512m drive disk1
   sd length 512m drive disk2
   sd length 512m drive disk3
 volume usr
  plex org raid5 491k
   sd length 0 drive disk1
   sd length 0 drive disk2
   sd length 0 drive disk3

And then you run:

# gvinum create gvinum.conf

This will create three gvinum RAID-5 arrays – for

/var

/usr

and

/tmp

. They will be accessible via

/dev/gvinum/var

/dev/gvinum/usr

and

/dev/gvinum/tmp

respectively. You should know, that the size of the RAID-5 array is the sum of the size of all drives – the size of one drive. This makes our

/var

2 GB,

/tmp

1GB and

/usr

the rest. If you execute ”

gvinum list

” now, you’ll see that all the arrays are marked as up. However, this will not be the case after you reboot. I don’t know exactly why, perhaps this is a bug. Also I am not sure if it is present in the newest FreeBSD releases. So it is very important now, that you reboot the system now. After it comes back online, you have to run:

# gvinum start var
# gvinum start usr
# gvinum start tmp

Then you have to wait for the arrays to become synchronized. It may take a while. You can always check status with ”

gvinum list

“. When the arrays are synchronized, you need to create the filesystems on all of them:

# newfs /dev/mirror/root
# newfs -U /dev/gvinum/var
# newfs -U /dev/gvinum/usr
# newfs -U /dev/gvinum/tmp

After that, you should mount these new arrays in

/mnt

and copy the system, you are running now onto them:

# mount /dev/mirror/root /mnt
# cd /mnt
# mkdir var tmp usr
# chmod 1777 tmp
# mount /dev/gvinum/var /mnt/var
# mount /dev/gvinum/usr /mnt/usr
# mount /dev/gvinum/tmp /mnt/tmp
# cd / && find . -xdev | cpio -pm /mnt
# cd /var && find . -xdev | cpio -pm /mnt/var
# cd /usr && find . -xdev | cpio -pm /mnt/usr
# cd /tmp && find . -xdev | cpio -pm /mnt/tmp

Finally, you have to modify your

fstab

file on the root gmirror array. Edit

/mnt/etc/fstab

as follows:

# Device                Mountpoint      FStype  Options         Dump    Pass#
/dev/mirror/swap        none            swap    sw              0       0
/dev/mirror/root        /               ufs     rw              1       1
/dev/gvinum/tmp         /tmp            ufs     rw              2       2
/dev/gvinum/usr         /usr            ufs     rw              2       2
/dev/gvinum/var         /var            ufs     rw              2       2
/dev/acd0               /cdrom          cd9660  ro,noauto       0       0

Now you can try to boot the system from one of the three drives which hold the RAID arrays and you should be lucky. If you aren’t, you are welcome to post comments here and we’ll try to sort it out together.

OpenVPN package generator

admin — Mon, 24 Nov 2008 09:42:30 +0000

OpenVPN is an open, secure and thoroughly tested VPN technology. Our company Infrax d.o.o. also provides OpenVPN clients for customers to be able to access their companies’ networks. We don’t use password-based authentication, but certificate-based one, which means we have our own full-blown Certificate Authority (self signed, of course). We use EJBCA for that and it works beautifully.

However, our customers are expecting simple solutions: a single package with their own config file and certificate. Although EJBCA supports generating OpenVPN packages on the fly on the server, it would be a bit more complicated to make it work, because we need to generate different config files for our customers. Which means we would need a special lookup-table (we could use the EJBCA’s MySQL database) which would link specific fields in certificates’ “distingished name” with corresponding OpenVPN config files. This would be possible, but a bit too complicated for now.

So I decided that we create the packages “semi-manually”, i.e. manually but in “batch” mode. First, you need my package which includes OpenVPN source environment (OpenVPN 2.1 RC7) which will be compiled with NSIS into an executable together with the certificate and config files.

Then you need to obtain the certificates for the clients you want to create the packages for and provide the config file as well. You must also install NSIS and then you are ready. Just run “generate.cmd” file in the “openvpn-generator” directory and the executables will be created in “openvpn-generator/exes”.

If you have problems, questions, just let me know.

Nejc Škoberne, univ. dipl. inž. rač. in inf.

admin — Fri, 21 Nov 2008 01:50:59 +0000

Voilà. I just graduated today (actually yesterday, it is early morning when I am writing this). A chapter of my life which lasted for the last 6 years is closed now. And it feels good.

Thanks to everybody who came to my presentation. It is very nice to have friends like you. Also thanks to my mentor, Borut Robič and to my parents who supported me for all those years. Thanks to mom, Janja and Anica who baked all the cookies for the short after-party in the LALG laboratory. Finally, special thanks to my wife Mica and my son Jakob, who felt a bit of stress in the last days too.

You can get my thesis here. It is titled “Improvement of pfSense firewall with user services” or “Nadgradnja požarnega zidu pfSense z uporabniškimi storitvami” (Slovenian translation). For non-Slovenian readers, the text is in Slovenian language, so probably you won’t understand most of it. However, I plan to translate at least the most important parts into English so it could be used by pfSense developers as well. I’ll make a special page called Papers soon, so all my papers will be there some day.

RCPFA 1.0.4 released

admin — Wed, 19 Nov 2008 09:28:33 +0000

Dear RoundCube and PostfixAdmin users, I just released the 1.0.4 version of RCPFA bridge, which makes RoundCube and PostfixAdmin interoperate. Many people have requested that it would be nice to have RCPFA for RoundCube 0.2beta available, and here it is. So this version doesn’t provide any new features, it only provides compatibility with 0.2beta. Hope it works for you too.