Answer for Python 3 : Run the following and restart your cluster

dbutils.fs.put("/databricks/init/gdal-initpython3.sh",""" #!/bin/bash export GDAL_VERSION=2.1.3+dfsg-1~xenial2 add-apt-repository ppa:ubuntugis/ppa && apt-get update apt-get install -y gdal-bin=$GDAL_VERSION python-gdal=$GDAL_VERSION libgdal-dev=$GDAL_VERSION libgdal-java=$GDAL_VERSION libopencv-dev python-opencv apt install gdal-bin python3-gdal """, True)

Hey guys you must put the gdal installation in a Init_Script of the cluster.

Because if you run this "sudo add-apt-repository -y ppa:ubuntugis/ubuntugis-unstable

sudo apt-get install gdal-bin python-gdal python3-gdal" or this "%sh sudo add-apt-repository -y ppa:ubuntugis/ppa ; sudo apt update -y ; sudo apt install -y gdal-bin python-gdal python3-gdal"

You are installing only on the Master Node and not on the worker nodes.

So steps I did and it work:

1- Create a folder where you want to put the init_script

My example: dbutils.fs.mkdirs("dbfs:/databricks/FileStore/init_script") ;

2- Create the init_script for the gdal- I called it gdal.sh

dbutils.fs.put("/databricks/FileStore/init_script/gdal.sh","""

#!/bin/bash

set -ex

/databricks/python/bin/python -V

. /databricks/conda/etc/profile.d/conda.sh

conda activate /databricks/python

conda install -y gdal""", True)

3- Then you must create a cluster to set the init_script (You can't set the init_script after the cluster is already built). To set the init_script you go to Advanced Options.

Then you restart the cluster and it is done!! :)

@ykb5t This worked for me:

%sh sudo add-apt-repository -y ppa:ubuntugis/ppa ; sudo apt update -y ; sudo apt install -y gdal-bin python-gdal python3-gdal

has there been a solution yet?

@emissary I have faced same issue. Resolved it by adding below lines at the end of bash script.

export CPLUS_INCLUDE_PATH=/usr/include/gdal

export C_INCLUDE_PATH=/usr/include/gdal

/databricks/python/bin/pip install gdal==2.1

Tried everything, unable to get it working on databricks runtime > 4 , it was working on 2.x? Does anyone have a solution to get GDAL working on databricks runtime 6.x?

We are having the same issue, trying to install GDAL on Databricks Runtime version 6.5.

We've added the following init script and created a cluster set up to run this.

#!/bin/bash
apt-get update -y
apt-get install -y software-properties-common
export GDAL_VERSION=2.2.2+dfsg-1~xenial1
add-apt-repository ppa:ubuntugis/ppa -y && apt-get update -y
apt-get install -y --allow-unauthenticated gdal-bin=$GDAL_VERSION python-gdal=$GDAL_VERSION libgdal-dev=$GDAL_VERSION libgdal-java=$GDAL_VERSION
apt-get install -y --allow-unauthenticated gdal-bin python3-gdal

We installed GDAL via the libraries import and ran the following in a new notebook.

from osgeo import gdal
gdal.VersionInfo('VERSION_NUM')

This resulted in the error:

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
/usr/lib/python3/dist-packages/osgeo/__init__.py in swig_import_helper()
     10         try:
---> 11             fp, pathname, description = imp.find_module('_gdal', [dirname(__file__)])
     12         except ImportError:/local_disk0/pythonVirtualEnvDirs/virtualEnv-ed414f1c-1a2c-4a29-9a4f-a9cd81d213ef/lib/python3.7/imp.py in find_module(name, path)
    295     else:
--> 296         raise ImportError(_ERR_MSG.format(name), name=name)
    297 ImportError: No module named '_gdal'During handling of the above exception, another exception occurred:ModuleNotFoundError                       Traceback (most recent call last)
<command-3699945019213047> in <module>
      7 
      8 import sys
----> 9 from osgeo import gdal
     10 
     11 gdal.VersionInfo('VERSION_NUM')/usr/lib/python3/dist-packages/osgeo/__init__.py in <module>
     19                 fp.close()
     20             return _mod
---> 21     _gdal = swig_import_helper()
     22     del swig_import_helper
     23 else:/usr/lib/python3/dist-packages/osgeo/__init__.py in swig_import_helper()
     11             fp, pathname, description = imp.find_module('_gdal', [dirname(__file__)])
     12         except ImportError:
---> 13             import _gdal
     14             return _gdal
     15         if fp is not None:ModuleNotFoundError: No module named '_gdal'

We have read that this error may be due to misaligned PATH and PYTHONPATH environment variables, but are unsure how to rectify this issue.

Any help would be much appreciated.

Hi all,

We have been having these issues also recently and last night I was able to successfully install GDAL (2.2.2) on our cluster.

Our cluster config (refined to include items of potential interest) is as follows:

{ "jdbc_port": 10000, "spark_version": "6.5.x-scala2.11", "node_type_id": "Standard_DS3_v2", "driver_node_type_id": "Standard_DS3_v2", "cluster_log_conf": { "dbfs": { "destination": "dbfs:/cluster-logs" } }, "spark_env_vars": { "PYSPARK_PYTHON": "/databricks/python3/bin/python3" }, "autotermination_minutes": 30, "enable_elastic_disk": true, "cluster_source": "UI", "init_scripts": [ { "dbfs": { "destination": "dbfs:/databricks/FileStore/init_script/gdal_install.sh" } } ], "init_scripts_safe_mode": false }

The init_script we have used:

#!/usr/bin/env bash

apt-get update -y # Needed to install add-apt-repository. apt-get install -y software-properties-common add-apt-repository ppa:ubuntugis/ppa -y && apt-get update -y

Install the necessary gdal binaries and libraries

Note: At the time of writing the latest version available is 2.2.2 for Ubuntu 16.04 (xenial) apt-get install -y gdal-bin=2.2.2+dfsg-1~xenial1 apt-get install -y libgdal-dev=2.2.2+dfsg-1~xenial1

Important: when pip installing GDAL, the location of the installed binary and library files must be setup

xport CPLUS_INCLUDE_PATH=/usr/include/gdal export C_INCLUDE_PATH=/usr/include/gdal

Finally, since gdal relies on numpy being installed, the installation of both gdal and numpy are performed here.

Also note the version of python being used is that of the one available to databricks and spark. /databricks/python/bin/pip install numpy /databricks/python/bin/pip install gdal==2.2.2

You will have to upload the init script to or, using the inbuilt dbutils library, create a file in the appropriate DBFS folder (see Cluster node initialization scripts) and setup the cluster to load the script at boot via the Advanced section when creating the cluster.

For more information on how to configure the cluster, see Cluster Configuration - Python.

Finally once the cluster is running, the installation of the python bindings to GDAL can be checked by creating a cell in a notebook with the following code:

# Check version of GDAL installed - should be 2020200 from osgeo import gdal print(f"GDAL Version: {gdal.VersionInfo('VERSION_NUM')}")

Create a POINT geometry

rom osgeo import ogr

point = ogr.Geometry(ogr.wkbPoint) point.AddPoint(1198054.34, 648493.09) print(f"Point created: {point.ExportToWkt()}")

GDAL Version: 2020200 Point created: POINT (1198054.34 648493.09 0)

If you don't get any errors running the above, you should now have GDAL at your disposal.

dbutils.fs.put("/databricks/init_scripts/gdal.sh","""

#!/usr/bin/env bash
apt-get update -y # Needed to install add-apt-repository. 
apt-get install -y software-properties-common
add-apt-repository ppa:ubuntugis/ppa -y && apt-get update -y
apt-get install -y gdal-bin=2.2.2+dfsg-1~xenial1
apt-get install -y libgdal-dev=2.2.2+dfsg-1~xenial1

""", True)