Import Python Functions In Databricks: A Quick Guide

Nov 10, 2025 by Admin 53 views

Hey everyone! Ever found yourself needing to reuse some awesome Python code you wrote in another file while working in Databricks? It's a super common scenario, and luckily, Databricks makes it pretty straightforward to import functions from other Python files. This guide will walk you through the different methods, step by step, so you can keep your code modular and organized. Let's dive in!

Why Import Functions?

Before we jump into the how, let's quickly touch on the why. Why bother importing functions at all? Well, imagine you've written a fantastic function for data cleaning. Do you want to copy and paste that code into every single notebook where you need it? Of course not! That would be a nightmare to maintain. Importing functions allows you to:

Keep your code DRY (Don't Repeat Yourself): Write your function once, use it everywhere.
Improve code organization: Separate your code into logical modules, making it easier to understand and maintain.
Enhance reusability: Share your functions across multiple projects.
Simplify debugging: When you fix a bug in one place, it's fixed everywhere the function is used.

So, importing functions is all about writing cleaner, more maintainable, and reusable code. Now, let's see how to do it in Databricks.

Method 1: Using `%run`

The %run magic command is one of the simplest ways to import functions from another Python file within a Databricks notebook. It essentially executes the specified Python file as if its contents were directly pasted into your current notebook. This makes any functions defined in that file immediately available for use.

Here's how it works:

Create your Python file: Let's say you have a file named my_functions.py stored in the same directory as your notebook (we'll talk about other locations later). This file contains the functions you want to import. For example:
```
# my_functions.py
def greet(name):
    return f"Hello, {name}!"

def add(x, y):
    return x + y
```
Use %run in your notebook: In your Databricks notebook, simply use the %run magic command followed by the path to your Python file:
```
%run ./my_functions.py
```
Important Note: The ./ indicates that the file is in the current directory. We'll cover different path scenarios shortly.
Call your functions: Now you can directly call the functions defined in my_functions.py as if they were defined in your notebook:
```
print(greet("Alice"))  # Output: Hello, Alice!
result = add(5, 3)
print(result)         # Output: 8
```

Explanation:

The %run command essentially takes the code from my_functions.py and executes it within the current notebook's scope. This makes the greet and add functions directly accessible. It's a quick and easy way to import functions, especially for simple projects or when you're just experimenting.

Important Considerations with %run:

Scope: Be mindful of variable names. If my_functions.py defines a variable with the same name as one in your notebook, the value in my_functions.py will overwrite the existing one.
Execution: %run executes the entire script. If your my_functions.py has any top-level code outside of function definitions (e.g., printing statements), that code will also be executed when you use %run.

Method 2: Using `import` and Modules

A more structured and recommended approach is to use Python's import statement. This allows you to treat your Python file as a module and import specific functions (or all functions) from it. This method offers better control and organization compared to %run.

Here's how it works:

Create your Python file (same as before): Assume you have my_functions.py with the greet and add functions.
Ensure the file is accessible: This is where things get a bit more nuanced. Python needs to be able to find your my_functions.py file. There are a few common scenarios:
- File in the same directory: If my_functions.py is in the same directory as your notebook, you can simply use:
```
import my_functions
```
- File in a different directory: If the file is in a different directory, you need to tell Python where to look. There are a couple of ways to do this:
  - Append to sys.path: You can add the directory containing your file to Python's search path using the sys module:
```
import sys
sys.path.append("/path/to/your/directory")
import my_functions
```
    Replace "/path/to/your/directory" with the actual path to the directory containing my_functions.py. Important: This change is temporary and only applies to the current notebook session.
  - Install as a Package (Best Practice): For more complex projects, the best approach is to package your code into a proper Python package. This involves creating a setup.py file and installing the package using pip. This makes your code easily reusable across multiple projects and environments. This is beyond the scope of this quick guide, but definitely worth learning for larger projects. Look into setuptools documentation for this purpose.
Import and use your functions: Once Python can find your module, you can import and use your functions in a few ways:
- Import the entire module:
```
import my_functions
print(my_functions.greet("Bob"))  # Output: Hello, Bob!
result = my_functions.add(10, 7)
print(result)         # Output: 17
```
  Here, you access the functions using the module name (my_functions) followed by a dot and the function name.
- Import specific functions:
```
from my_functions import greet, add
print(greet("Charlie"))  # Output: Hello, Charlie!
result = add(20, 5)
print(result)         # Output: 25
```
  This imports only the greet and add functions directly into your notebook's namespace, allowing you to call them without the my_functions. prefix.
- Import with an alias:
```
from my_functions import greet as say_hello
print(say_hello("David"))  # Output: Hello, David!
```
  This imports the greet function but gives it a different name (say_hello) in your notebook. This can be useful to avoid naming conflicts or to provide a more descriptive name.

Advantages of using import:

Organization: Modules provide a clear structure for your code.
Namespaces: Using import my_functions creates a separate namespace, reducing the risk of naming conflicts.
Readability: The code is often more readable and easier to understand.
Best Practice: This is the standard way to import code in Python.

Method 3: Using Databricks Utilities (dbutils)

Databricks provides a set of utilities called dbutils that offer various functionalities, including file system manipulation. While not directly for importing functions, dbutils.fs.cp can be used to copy Python files into the working directory of your Databricks notebook, after which you can use %run or import as described above.

Here’s the process:

Store your Python file in Databricks File System (DBFS): First, you need to upload your Python file (e.g., my_functions.py) to DBFS. You can do this through the Databricks UI or using dbutils.fs.put.
```
dbutils.fs.put("/dbfs/my_functions.py", """def greet(name):
    return f'Hello, {name}!'

def add(x, y):
    return x + y""", overwrite = True)
```
Copy the file to the local file system: Use dbutils.fs.cp to copy the file from DBFS to the local file system of the Databricks cluster. This will typically be a temporary directory.
```
dbutils.fs.cp("dbfs:/my_functions.py", "file:/tmp/my_functions.py")
```
Import the functions: Now that the file is in the local file system, you can use %run or import to import the functions:
```
%run /tmp/my_functions.py
# or
import sys
sys.path.append("/tmp")
import my_functions
```

When to use dbutils:

Accessing files stored in DBFS: This method is useful when your Python files are stored in DBFS and you need to make them available to your notebook.
Dynamically loading files: If you need to load different files based on certain conditions, you can use dbutils to copy the appropriate file to the local file system before importing it.

Caveats:

Complexity: This method is more complex than simply using %run or import directly.
Temporary Files: The copied file exists only for the duration of the Databricks session in the /tmp directory.

Choosing the Right Method

So, which method should you use? Here's a quick summary:

%run: Easiest for simple scripts and quick experiments, especially when the file is in the same directory. Be mindful of scope and execution of the entire script.
import: The recommended approach for most scenarios. Provides better organization, namespaces, and readability. Requires ensuring the file is accessible by Python (e.g., by adding it to sys.path or installing it as a package).
dbutils: Useful when dealing with files stored in DBFS or when you need to dynamically load files. More complex than the other methods.

In general, start with import if you can. It's the most robust and Pythonic way to manage your code. If you need a quick and dirty solution for a simple script, %run might suffice. And if you're working with files in DBFS, dbutils can be helpful.

Best Practices

Organize your code: Use modules and packages to structure your code logically.
Use meaningful names: Give your functions and variables descriptive names.
Write documentation: Add docstrings to your functions to explain what they do and how to use them.
Test your code: Write unit tests to ensure your functions are working correctly.
Consider using Repos: Databricks Repos provides a great way to manage code, integrate with Git, and ensure version control.

Conclusion

Importing functions from other Python files in Databricks is crucial for writing clean, maintainable, and reusable code. Whether you choose %run, import, or dbutils, understanding these methods will significantly improve your development workflow. So go ahead, start organizing your code, and make your Databricks notebooks shine! Happy coding, guys!