Webpage Screenshot Server

How to build a Webpage Screenshot Server with the Raspberry Pi 2. First we need "xvfb" (X virtual framebuffer), ImageMagick then we need "firefox" which is called "Iceweasel" for the Raspberry Pi and last we need "apache" with php to listen on external requests and deliver the Webpage Screenshots. Many tanks to Leonard Teo (Taking Server Side Screenshots of Websites) I followed his instructions for Ubuntu and adapted them for the Raspberry Pi.

1.) Update your Raspberry Pi Packages

run the commands:

pi@raspberrypi ~ $ sudo apt-get update
pi@raspberrypi ~ $ sudo apt-get upgrade

2.) Install xvfb, iceweasel (Firefox), imagemagick, apache2, php5

run the commands:

pi@raspberrypi ~ $ sudo apt-get install xvfb
pi@raspberrypi ~ $ sudo apt-get install iceweasel
pi@raspberrypi ~ $ sudo apt-get install imagemagick
pi@raspberrypi ~ $ sudo apt-get install apache2
pi@raspberrypi ~ $ sudo apt-get install php5 php5-gd libapache2-mod-php5

to remove unused packages run the command:

pi@raspberrypi ~ $ sudo apt-get --purge remove sonic-pi
pi@raspberrypi ~ $ sudo apt-get autoremove

3.) Check the Package "ttf-mscorefonts-installer"

Many webpages use common fonts that are included in the "ttf-mscorefonts-installer" package, check if it is already installed or you might want to install it.

pi@raspberrypi ~ $ dpkg --get-selections
pi@raspberrypi ~ $ sudo apt-get install ttf-mscorefonts-installer

4.) Get the "xvfb" process running

create a file in /etc/init.d called xvfb with te following shell script:

#! /bin/sh
### BEGIN INIT INFO
# Provides:          xvfb
# Required-Start:    $remote_fs $syslog
# Required-Stop:     $remote_fs $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Start/stop xvfb for the Screenserver
### END INIT INFO

# 11.03.2017 david jufer

XVFB_OUTPUT=/tmp/Xvfb.out
XVFB=/usr/bin/X11/Xvfb
XVFB_OPTIONS=":5 -screen 0 1920x1920x24 -fbdir /var/run"

do_start()
{
    echo -n "Starting : X Virtual Frame Buffer "
    $XVFB $XVFB_OPTIONS >>$XVFB_OUTPUT 2>&1&
    RETVAL=$?
    echo
    return $RETVAL
}

do_stop()
{
    echo -n "Shutting down : X Virtual Frame Buffer"
    echo
    killall Xvfb
    echo
    return 0
}

case "$1" in
    start)
        do_start
    ;;
    stop)
        do_stop
    ;;
    status)
        status xvfb
    ;;
    restart)
        do_stop
        do_start
    ;;
    *)
        echo "Usage: xvfb {start|stop|status|restart}"
        exit 1
    ;;
esac

:

now you can start and stop the xvfb server with:

pi@raspberrypi ~ $ sudo service xvfb start
pi@raspberrypi ~ $ sudo service xvfb stop

with the following command we achiev that the process "xvfb" starts automatically on startup (on CentOS it's "chkconfig"):

pi@raspberrypi ~ $ sudo service --status-all
pi@raspberrypi ~ $ sudo update-rc.d xvfb defaults

5.) Check the basic functions

Let us check the basic functions. Start the virtual framebuffer, run the Firefox-Browser an take a screenshot of it. DISPLAY=:5 tells xvfb to render to display 5 (virtual), nohup silence the output firefox loads, firefox jufer.info starts Iceweasel and loads the given URL.

pi@raspberrypi ~ $ sudo service xvfb start
pi@raspberrypi ~ $ sudo DISPLAY=:5 nohup firefox jufer.info &
pi@raspberrypi ~ $ sudo DISPLAY=:5 import -window root /var/www/html/screenshot.png

Check the PNG-Image. Kill firefox by typing fg then CTRL+C.

6.) Some Config for Apache

Setup the rewrite module and configuration overriding (.htaccess)

pi@raspberrypi ~ $ sudo a2enmod rewrite
pi@raspberrypi ~ $ sudo nano /etc/apache2/sites-enabled/000-default.conf

change the settings for "Directory /var/www/" to look like this

<VirtualHost *:80>
  ServerAdmin webmaster@localhost
  DocumentRoot /var/www/html
  # david jufer 11.03.2017
  <Directory /var/www/html>
    AllowOverride All
  </Directory>
  ErrorLog ${APACHE_LOG_DIR}/error.log
  CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>

Restart Apache

pi@raspberrypi ~ $ sudo service apache2 restart

7.) Some Config for Firefox/Iceweasel

After closing firefox with "kill %1" the profile ist writen to /root/.mozilla/firefox/<random>.default/

now check /root/.mozilla/firefox/profiles.ini to look something like this

pi@raspberrypi ~ $ sudo nano /root/.mozilla/firefox/profiles.ini

[General]
StartWithLastProfile=0

[Profile0]
Name=default
IsRelative=1
Path=sa8a1kyt.default
Default=1

open the file "prefs.js"

pi@raspberrypi ~ $ nano /root/.mozilla/firefox/sa8a1kyt.default/prefs.js

add the folowing content at the end of the file:

# david jufer 11.03.2017
user_pref("browser.fullscreen.animateUp", 0);
user_pref("browser.fullscreen.autohide", false);
user_pref("browser.link.open_newwindow", 1);
user_pref("browser.sessionstore.resume_from_crash", false);
user_pref("browser.startup.page", 0);
user_pref("browser.tabs.warnOnOpen", false);

open /root/.mozilla/firefox/<random>.default/xulstore.json an set the following values for with, height, screenX, screenY

pi@raspberrypi ~ $ sudo nano /root/.mozilla/firefox/sa8a1kyt.default/xulstore.json

...
,"main-window":{"width":"1920", "height":"1920", "screenX":"0", "screenY":"0",
...

8.) Create the scripts needed on the Apache

We want the URL-Call to be something like this to create a Image of the Webpage www.jufer.info:
   http://domain/screenserver/www.jufer.info
or
   http://domain/screenserver/1280/1800/www.jufer.info
to get a screenshot of the size 1280x1800.
Fist crate a folder /var/www/html/screenserver

pi@raspberrypi ~ $ sudo mkdir /var/www/html/screenserver
pi@raspberrypi ~ $ sudo chown www-data:root /var/www/html/screenserver

Create the 2 empty files jobKey.txt and jobUrl.txt make shure the owner is "www-data". With these 2 files we are achieving that multiple requests are serialized.

Create the /screenserver/.htaccess file with the following rules:

RewriteEngine on

RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^([0-9]+)/([0-9]+)/(.+)[/]?$ index.php?url=$3&width=$1&height=$2 [PT]

RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^([0-9]+)/(.+)[/]?$ index.php?url=$2&width=$1 [PT]

RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^(.+)[/]?$ index.php?url=$1 [PT]

The index.php file contains the function of the "frontend" an look like this:

http://192.168.1.27/screenserver/www.php.net output png 1920x1920 pixel

http://192.168.1.27/screenserver/800/raspberrypi.org output png 800x600 pixel

http://192.168.1.27/screenserver/600/600/www.jufer.info output png 600x600 pixel

output png of raspberrypi.org 1280x1800 pixel

/var/www/html/screenserver/.htaccess

/var/www/html/screenserver/index.php

/root/screenserver.sh

/etc/cron.d/screenserver

download
index.php
index.pdf (27.6 KB)

9.) Setting up the Server-Side

Until now we recive jobs but the call to firefox and the ImageMagick part is missing. This is done with a shell-script. Additional we need some setting to start the server-script automaticaly on reboot and some cleanup jobs.

Crate a shell-script in /root/screenserver.sh

# file: screenserver.sh
# date: 11.03.2017
# author: david jufer

:>/var/www/html/screenserver/jobKey.txt
:>/var/www/html/screenserver/jobUrl.txt

while [ 1 ]
do
  URL=`less /var/www/html/screenserver/jobUrl.txt`
  KEY=`less /var/www/html/screenserver/jobKey.txt`
  STR=""
  # if todo is not empty - do something
  if [ "$URL" != "$STR" ]; then
    # open url in firefox, wait
    DISPLAY=:5 firefox $URL &
    sleep 25
    # grab screen to png
    FILEIMGTMP="/var/www/html/screenserver/__$KEY.png"
    FILEIMGEND="/var/www/html/screenserver/_$KEY.png"
    DISPLAY=:5 import -window root $FILEIMGTMP
    # change filename
    mv $FILEIMGTMP $FILEIMGEND
    # wait
    sleep 1
    # reset job files
    :>/var/www/html/screenserver/jobKey.txt
    :>/var/www/html/screenserver/jobUrl.txt
  else
    sleep 1
  fi
done

Set the chmod for execution:

pi@raspberrypi ~ $ sudo chmod 755 /root/screenserver.sh

Get the script startet on reboot and do some cleanup:

pi@raspberrypi ~ $ sudo nano /etc/cron.d/screenserver

# /etc/cron.d/xvfb
# start screenserver script
# 11.03.2017 david jufer

# start xvfb on startup
@reboot root /root/screenserver.sh > /dev/null
@reboot root rm -f /var/www/html/screenserver/_*
15 0 * * * root rm -f /var/www/html/screenserver/_*

Congratulations you're done!

Reboot you're Raspberry Pi an check if it is working (my Raspberry Pi has the IP 192.168.1.27):
   http://192.168.1.27/screenserver/raspberrypi.org
after ca. 30s you should see the generated Screenshot-PNG

Debugging

Check the LOG files, check file rights and ownership...

> less /var/log/apache/error.log

> less /var/log/messages