DataStore plugin

The CKAN DataStore extension provides an ad hoc database for storage of structured data from CKAN resources.

Hint

Ref info page at http://docs.ckan.org/en/ckan-2.5.2/maintaining/datastore.html

Database

Create database user and a DB for the datastore:

su - postgres -c "createuser -S -D -R -P -l datastore"
su - postgres -c "createdb -O ckan datastore -E utf-8"

Make sure you also created the ckan DB user as documented in PostgreSQL configuration.

Configuration file

Open the file /etc/ckan/default/production.ini and add the datastore plugin:

ckan.plugins = datastore [... other plugins...]

Also set up the access to the datastore DB:

ckan.datastore.write_url = postgresql://ckan:PASSWORD@localhost/datastore
ckan.datastore.read_url = postgresql://datastore:PASSWORD@localhost/datastore

Database grants

CKAN needs to change some grants on the datastore.

CKAN can create the SQL script file, that shall be then run with postgres privileges.

Create the file as the ckan user:

cd
. default/bin/activate
paster --plugin=ckan datastore set-permissions -c /etc/ckan/default/production.ini > set_permissions.sql

Then execute the .sql file as postgres; or, as root, run:

su - postgres -c "psql  postgres -f /usr/lib/ckan/set_permissions.sql"

Restart CKAN to enable the datastore plugin:

systemctl restart supervisord

Then perform a ruequest to your server to make sure the datastore plugin replies properly:

http://YOUR_SITE/api/3/action/datastore_search?resource_id=_table_metadata

File storage plugin

FileStore allows users to upload data files to CKAN resources, and to upload logo images for groups and organizations.

Hint

Ref info page at http://docs.ckan.org/en/ckan-2.5.2/maintaining/filestore.html

FileStore is used to enable data upload in CKAN.

Create directory

mkdir -p /var/lib/ckan/upload
chown ckan: -R /var/lib/ckan

Set the storage config in production.ini:

ckan.storage_path = /var/lib/ckan/upload

DataPusher

Automatically add Data to the CKAN DataStore.

Hint

Doc page at http://docs.ckan.org/projects/datapusher/en/latest/index.html

As root install the WSGI apache module:

yum install mod_wsgi

As ckan, create a brand new virtualenv, and install the datapusher app in it:

virtualenv /usr/lib/ckan/datapusher
mkdir /usr/lib/ckan/datapusher/src
cd /usr/lib/ckan/datapusher/src
git clone -b stable https://github.com/ckan/datapusher.git
cd datapusher/
. ../../bin/activate
pip install setuptools==36.1
pip install -r requirements.txt
python setup.py develop

Create configuration files:

cp /usr/lib/ckan/datapusher/src/datapusher/deployment/datapusher_settings.py /etc/ckan/default/datapusher_settings.py

cp /usr/lib/ckan/datapusher/src/datapusher/deployment/datapusher.wsgi /etc/ckan/default/datapusher.wsgi

Then edit /etc/ckan/default/datapusher.wsgi and adjust the settings path from:

os.environ['JOB_CONFIG'] = '/etc/ckan/datapusher_settings.py'

to

os.environ['JOB_CONFIG'] = '/etc/ckan/default/datapusher_settings.py'

Then create a file name /etc/httpd/conf.d/94-datapusher.conf and add these lines:

Listen 8800

<VirtualHost 0.0.0.0:8800>

   ServerName ckan

   # this is our app
   WSGIScriptAlias / /etc/ckan/default/datapusher.wsgi

   # pass authorization info on (needed for rest api)
   WSGIPassAuthorization On

   # Deploy as a daemon (avoids conflicts between CKAN instances)
   WSGIDaemonProcess datapusher display-name=demo processes=1 threads=15

   WSGIProcessGroup datapusher

   ErrorLog /var/log/httpd/datapusher.error.log
   CustomLog /var/log/httpd/datapusher.log combined

   <Directory "/" >
      Require all granted
   </Directory>

</VirtualHost>

Now let’s allow connections to port 8800 in SELinux:

semanage port -a -t http_port_t -p tcp 8800

and restart httpd in order to load the new configuration:

systemctl restart httpd

Test the datapusher entrypoint with a request like

curl http://localhost:8800

on the same machine ckan is running on. You should get a response like this:

{
  "help": "\n        Get help at:\n        http://ckan-service-provider.readthedocs.org/."
}

Now let’s make ckan aware that the datapusher is available.

Edit the file /etc/ckan/default/production.ini and:

• add the datapusher plugin:

  ckan.plugins = [... other plugins...] datapusher

• remove the comments from the lines:

  ckan.datapusher.formats = csv xls xlsx tsv application/csv application/vnd.ms-excel application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
  ckan.datapusher.url = http://127.0.0.1:8800/

Eventually restart supervisord to make ckan reload the configuration:

systemctl restart supervisord