openGrok Applicance
Wednesday, June 4th, 2008
OpenGrok is an open source soure code browser.
It creates and index over code and allows you to quickly find and browse the annotated source.
To use it, you’d need a java webserver, and of course some code to run it on
OpenGrok’s creators use it for the Open Solaris Source - incidentially a good example - try searching for the defintion of strcmp.
OpenGrok isn’t difficult to set up - if you have a running Tomcat with a Java runtime 1.5, that is. Now, that’s a bit unussual for the shop I work with, so I decided to implement a central code browser using VMWare (Player) and a virtual Debian.
Step 1: Get yourself a Debian vm
Either download the vm found on Vmware’s VM Applicances site, or get the Debian Netinstall. I went with the Netinstall (and VBX Creator), since I wanted my VM to have two virtual harddrives, one for the Debian system and one for our code. This will allow me easily to replace the code harddisk later on with a bigger one (this turned out to be necessary a few years later).
After answering the basic installations questions ( manual package selection, nothing ), I suggest apt-get install ssh, so you can access the vm from your favourite SSH client - I use Putty when on windows.
Step 2: Uprade to Debian Etch
Since Tomcat 5 isn’t in the current stable Debian (sarge) we’ll switch to testing (Etch) instead.
#first, my favourite editor…
apt-get install joe
joe /etc/apt/sources.list
add contrib…
Step 3: Java
Now, power off the VM and make a backup. During the package creation for Java, you’ll save some largish files that will enlarge your VM. I’ll only sketch the step here.
First, we’ll need the most current Java Runtime.
#Webbrowser
apt-get install w3m
w3m http://java.sun.com/javase/downloads/index.jsp
Fetch the java se runtime
Now, we’ll create an installable java package
apt-get install java-package
su nonRootUser
fakeroot fakeroot make-jpkg jre-1_5_0_06-linux-i586.bin
(obviously change the file name for your java runtime)
SCP the created .deb file to a different machine, throw away the VM and start your backup.
Install the .dev here.
Step 4: Install Tomcat
apt-get install tomcat 5
Step 5: Fetch and install openGrok
Download openGrok and move the source.jar to /var/lib/tomcat5/webapps
Move openGrok.jar and lib to /opt/openGrok
Restart Tomcat (/etc/init.d/tomcat 5 restart )
Step 6: Mount the second harddrive
fdisk /dev/hdb # one partition is enough
mkfs.ext3 /dev/hdb5
joe /etc/fstab
…
mount /var/lib/tomcat5/webapps/source/code/
mkdir /var/lib/tomcat5/webapps/source/code/src
mkdir /var/lib/tomcat5/webapps/source/code/data
Step 6: Configure Tomcat
If your’re paranoid like me, Tomcat Apps need explicit permissions.
So place the following in your /etc/tomcat5/policies/04webapps.policy
grant codeBase “file:${catalina.home}/webapps/source/-” {
 permission java.util.PropertyPermission “subversion.native.library”, “read”;
permission java.lang.RuntimePermission “loadLibrary.svnjavahl-1″;
 permission java.lang.RuntimePermission “loadLibrary.libsvnjavahl-1″;
 permission java.lang.RuntimePermission “loadLibrary.svnjavahl”;
 permission java.util.PropertyPermission “disableLuceneLocks”, “read”;
 permission java.util.PropertyPermission “catalina.home”, “read”;
permission java.util.PropertyPermission “java.io.tmpdir”, “read”;
 permission java.util.PropertyPermission “org.apache.lucene.lockdir”, “read”;
 permission java.util.PropertyPermission “org.apache.lucene.writeLockTimeout”, “read”;
 permission java.util.PropertyPermission “org.apache.lucene.commitLockTimeout”, “read”;
 permission java.util.PropertyPermission “org.apache.lucene.mergeFactor”, “read”;
 permission java.util.PropertyPermission “org.apache.lucene.minMergeDocs”, “read”;
 permission java.util.PropertyPermission “org.apache.lucene.*”, “read”;
 permission java.io.FilePermission “/var/lib/tomcat5/temp”, “read”;
 permission java.io.FilePermission “/var/lib/tomcat5/temp/*”, “write”;
 permission java.io.FilePermission “/var/lib/tomcat5/temp/*”, “delete”;
};
grant codeBase “file:${catalina.home}/webapps/source/WEB-INF/lib/-” {
 permission java.util.PropertyPermission “subversion.native.library”, “read”;
 permission java.lang.RuntimePermission “loadLibrary.svnjavahl-1″;
 permission java.util.PropertyPermission “disableLuceneLocks”, “read”;
permission java.util.PropertyPermission “catalina.home”, “read”;
permission java.util.PropertyPermission “java.io.tmpdir”, “read”;
};
grant codeBase “file:${catalina.home}/webapps/source/WEB-INF/classes/-” {
permission java.util.PropertyPermission “subversion.native.library”, “read”;
 permission java.lang.RuntimePermission “loadLibrary.svnjavahl-1″;
 permission java.util.PropertyPermission “disableLuceneLocks”, “read”;
 permission java.util.PropertyPermission “catalina.home”, “read”;
 permission java.util.PropertyPermission “java.io.tmpdir”, “read”;
};
Restart Tomcat
Step 7: Fetch code
I use python and subversion (command line) to fetch our repositories each night, but then I already had a webbased repository list at hand.
apt-get install python subversion
Basically, I just call svn update on each working copy (or delete it and do a new checkout if that fails for some reason).
I could be using pysvn, but I never got it to work with Umlauts in the svn repository. Apperantly it simply ignores the locale I set…
Step 8: Create a shellscript and a cron
joe /opt/fetchAndIndexCode.sh
python /opt/fetchCode.py
java -jar /opt/openGrok.jar -s //var/lib/tomcat5/webapps/source/code/src /var/lib/tomcat5/webapps/source/code/data
chmod +x fetchAndIndexCode.sh
crontab -e
Step 9:
Run fetchAndIndexCode
Step 10: Visit http://yourip:8888/source
Step 11: Make tomcat accessible via Port 80
This is optional - for something more secure, I suggest using Apache and mod_proxy
I’d love to make it accessible from /, but apperantly there are some hardcoded paths in the webapp - and it’s not annyoing enough for me to work on that. I simple placed a meta redirect in /var/lib/tomcat5/webapps/ROOT/index.html
That’s it, I hope this quick guide has helped you.
I’d also love to get some comments, expieriences, improvements, etc.
So long,
Flo
OpenGrok is an open source soure code browser.
It creates and index over code and allows you to quickly find and browse the annotated source.
To use it, you’d need a java webserver, and of course some code to run it on
OpenGrok’s creators use it for the Open Solaris Source - incidentially a good example - try searching for the defintion of strcmp.
OpenGrok isn’t difficult to set up - if you have a running Tomcat with a Java runtime 1.5, that is. Now, that’s a bit unussual for the shop I work with, so I decided to implement a central code browser using VMWare (Player) and a virtual Debian.
Step 1: Get yourself a Debian vm
Either download the vm found on Vmware’s VM Applicances site, or get the Debian Netinstall. I went with the Netinstall (and VBX Creator), since I wanted my VM to have two virtual harddrives, one for the Debian system and one for our code. This will allow me easily to replace the code harddisk later on with a bigger one (this turned out to be necessary a few years later).
After answering the basic installations questions ( manual package selection, nothing ), I suggest apt-get install ssh, so you can access the vm from your favourite SSH client - I use Putty when on windows.
Step 2: Uprade to Debian Etch
Since Tomcat 5 isn’t in the current stable Debian (sarge) we’ll switch to testing (Etch) instead.
#first, my favourite editor…
apt-get install joe
joe /etc/apt/sources.list
add contrib…
Step 3: Java
Now, power off the VM and make a backup. During the package creation for Java, you’ll save some largish files that will enlarge your VM. I’ll only sketch the step here.
First, we’ll need the most current Java Runtime.
#Webbrowser
apt-get install w3m
w3m http://java.sun.com/javase/downloads/index.jsp
Fetch the java se runtime
Now, we’ll create an installable java package
apt-get install java-package
su nonRootUser
fakeroot fakeroot make-jpkg jre-1_5_0_06-linux-i586.bin
(obviously change the file name for your java runtime)
SCP the created .deb file to a different machine, throw away the VM and start your backup.
Install the .dev here.
Step 4: Install Tomcat
apt-get install tomcat 5
Step 5: Fetch and install openGrok
Download openGrok and move the source.jar to /var/lib/tomcat5/webapps
Move openGrok.jar and lib to /opt/openGrok
Restart Tomcat (/etc/init.d/tomcat 5 restart )
Step 6: Mount the second harddrive
fdisk /dev/hdb # one partition is enough
mkfs.ext3 /dev/hdb5
joe /etc/fstab
…
mount /var/lib/tomcat5/webapps/source/code/
mkdir /var/lib/tomcat5/webapps/source/code/src
mkdir /var/lib/tomcat5/webapps/source/code/data
Step 6: Configure Tomcat
If your’re paranoid like me, Tomcat Apps need explicit permissions.
So place the following in your /etc/tomcat5/policies/04webapps.policy
grant codeBase “file:${catalina.home}/webapps/source/-” {
 permission java.util.PropertyPermission “subversion.native.library”, “read”;
permission java.lang.RuntimePermission “loadLibrary.svnjavahl-1″;
 permission java.lang.RuntimePermission “loadLibrary.libsvnjavahl-1″;
 permission java.lang.RuntimePermission “loadLibrary.svnjavahl”;
 permission java.util.PropertyPermission “disableLuceneLocks”, “read”;
 permission java.util.PropertyPermission “catalina.home”, “read”;
permission java.util.PropertyPermission “java.io.tmpdir”, “read”;
 permission java.util.PropertyPermission “org.apache.lucene.lockdir”, “read”;
 permission java.util.PropertyPermission “org.apache.lucene.writeLockTimeout”, “read”;
 permission java.util.PropertyPermission “org.apache.lucene.commitLockTimeout”, “read”;
 permission java.util.PropertyPermission “org.apache.lucene.mergeFactor”, “read”;
 permission java.util.PropertyPermission “org.apache.lucene.minMergeDocs”, “read”;
 permission java.util.PropertyPermission “org.apache.lucene.*”, “read”;
 permission java.io.FilePermission “/var/lib/tomcat5/temp”, “read”;
 permission java.io.FilePermission “/var/lib/tomcat5/temp/*”, “write”;
 permission java.io.FilePermission “/var/lib/tomcat5/temp/*”, “delete”;
};
grant codeBase “file:${catalina.home}/webapps/source/WEB-INF/lib/-” {
 permission java.util.PropertyPermission “subversion.native.library”, “read”;
 permission java.lang.RuntimePermission “loadLibrary.svnjavahl-1″;
 permission java.util.PropertyPermission “disableLuceneLocks”, “read”;
permission java.util.PropertyPermission “catalina.home”, “read”;
permission java.util.PropertyPermission “java.io.tmpdir”, “read”;
};
grant codeBase “file:${catalina.home}/webapps/source/WEB-INF/classes/-” {
permission java.util.PropertyPermission “subversion.native.library”, “read”;
 permission java.lang.RuntimePermission “loadLibrary.svnjavahl-1″;
 permission java.util.PropertyPermission “disableLuceneLocks”, “read”;
 permission java.util.PropertyPermission “catalina.home”, “read”;
 permission java.util.PropertyPermission “java.io.tmpdir”, “read”;
};
Restart Tomcat
Step 7: Fetch code
I use python and subversion (command line) to fetch our repositories each night, but then I already had a webbased repository list at hand.
apt-get install python subversion
Basically, I just call svn update on each working copy (or delete it and do a new checkout if that fails for some reason).
I could be using pysvn, but I never got it to work with Umlauts in the svn repository. Apperantly it simply ignores the locale I set…
Step 8: Create a shellscript and a cron
joe /opt/fetchAndIndexCode.sh
python /opt/fetchCode.py
java -jar /opt/openGrok.jar -s //var/lib/tomcat5/webapps/source/code/src /var/lib/tomcat5/webapps/source/code/data
chmod +x fetchAndIndexCode.sh
crontab -e
Step 9:
Run fetchAndIndexCode
Step 10: Visit http://yourip:8888/source
Step 11: Make tomcat accessible via Port 80
This is optional - for something more secure, I suggest using Apache and mod_proxy
I’d love to make it accessible from /, but apperantly there are some hardcoded paths in the webapp - and it’s not annyoing enough for me to work on that. I simple placed a meta redirect in /var/lib/tomcat5/webapps/ROOT/index.html
That’s it, I hope this quick guide has helped you.
I’d also love to get some comments, expieriences, improvements, etc.
So long,
Flo